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1.0  Executive  Summary 

The  University  of  Memphis  conducted  basic  research  into  techniques  for  advancing 
network-centric  sensors  for  eventual  deployment  in  Department  of  Defense  (DoD) 
applications.  This  basic  research  included  the  following  focus  sub-areas:  i)  feature 
fusion/feature-based  sensor  system  design  techniques;  ii)  sensor  ontologies  for  problem¬ 
solving  architectures;  iii)  profiling  sensor  improvement  through  the  use  of  innovative 
classification  algorithms  and  data  visualization  techniques;  iv)  alternative  sensing 
modalities;  v)  turbulence  mitigation  techniques;  and  vi)  development  of  a  feature  sensing 
laboratory. 

Under  the  topic  of  feature  fusion/feature-based  sensor  system  design,  techniques  known 
as  Lasso,  Group  Lasso,  and  Sparse  Multiple  Kernel  Learning  were  applied  to  break  beam 
profiling  sensor  design.  The  results  indicate  that  the  Group  Lasso  technique  is  effective 
for  feature  quality  maximizing  sensor  design  because  of  its  ability  to  provide  both  inter¬ 
group  and  intra-group  feature  sparsity. 

Under  the  topic  of  sensor  ontologies  for  problem-solving  architectures,  a  framework  that 
matches  sensors  to  compatible  algorithms  to  fonn  synthesized  systems  was  developed 
and  applied  to  improved  forms  of  the  beam-break  profiling  sensor.  This  work  resulted  in 
several  publications. 

Under  the  topic  of  profiling  sensor  improvement,  various  algorithms  for  improving  the 
classification  performance  of  a  pyro-electric  based  profiling  sensor  were  investigated  and 
tested  using  data  from  field  collections.  Results  indicated  that  Logistic  regression  with  a 
simple  height  to  width  ratio  provide  good  performance. 

Under  the  topic  of  alternative  sensing  modalities,  a  model  for  micro- resonator  based 
sensors  was  developed,  research  toward  the  improvement  and  refinement  of  a  novel 
technique  for  forming  terahertz  images  was  performed,  and  the  use  of  compressive 
sensing  in  a  profile  type  detector  was  investigated.  This  work  resulted  in  several 
publications. 

Under  the  topic  of  turbulence  mitigation  techniques,  lucky  imaging,  blind  deconvolution, 
spectral  techniques,  and  frame  averaging  were  examined  using  turbulence  simulation 
against  their  ability  to  perform  in  a  wide  variety  of  tactical  situations.  The  strengths  and 
weaknesses  of  each  were  discussed  in  a  publication.  In  addition,  the  applicability  of 
turbulence  mitigation  techniques  to  sparse  array  sensors  such  as  profiling  sensors  was 
investigated. 

Under  the  topic  of  a  feature  sensing  laboratoiy,  the  design  and  implementation  of  an 
infrastructure  to  support  long-term  sensor  network  emplacements  in  a  laboratoiy  like 
environment  was  pursued.  To  date,  key  features  of  the  design  have  been  implemented 
with  final  integration  of  the  initial  deployment  likely  to  happen  in  September  2012. 


The  execution  of  this  contract  has  resulted  in  some  significant  scientific  and  technical 
progress  in  support  of  intelligent,  network-centric,  sensors.  We  believe  that  this  work  has 
been  of  immediate  benefit  to  the  Army  and  will  continue  to  pay  dividends  in  technical 
achievement  into  the  near  future.  Further  publications  will  result  directly  and  indirectly 
from  this  work. 


2.0  Introduction 

The  University  of  Memphis  conducted  basic  research  on  the  subject  of  Intelligent 
Network  Sensors  in  support  of  US  Army  basic  research  needs  in  the  areas  of  perimeter 
security  and  force  protection.  This  research  covered  a  broad  range  of  research  topics  and 
has  resulted  in  several  conference  and  journal  publications. 

The  topics  of  research  pursued  under  this  effort  were:  i)  feature  fusion/feature-based 
sensor  system  design  techniques;  ii)  sensor  ontologies  for  problem-solving  architectures; 
lii)  profiling  sensor  improvement  through  the  use  of  innovative  classification  algorithms 
and  data  visualization  techniques,  iv)  alternative  sensing  modalities;  v)  turbulence 
mitigation  techniques;  and  vi)  development  of  a  feature  sensing  laboratory.  A  detailed 
description  of  each  research  topic  pursued  in  this  effort  is  given  below.  Further  details 
and  copies  of  published  papers  may  be  found  in  the  Appendices. 

3.0  Feature  Fusion/Feature  Based  Sensor  Design 

Task  3.1 :  Demonstrate  sensor  optimization  based  on  enhancement  of 
feature  quality. 

Task  3.2:  Develop  a  feature  fusion  quality  metric  to  serve  as  a  cost  function 
for  optimizing  sensor  design. 

To  demonstrate  the  procedure  to  accomplish  the  above  stated  objectives,  the  design  of  a 
sparse  trip  wire  sensor  for  human  versus  animal  classification  is  used  as  a  case  study.  The 
sensor  consists  of  two  poles  that  can  be  placed  on  either  side  of  a  bottleneck  along  a  trail. 
One  pole  is  lined  with  NIR  transmitters  and  receivers,  while  the  other  pole  is  lined  with 
reflectors.  As  an  object  passes  between  the  two  poles,  it  blocks  the  path  between  the 
trans-receiver  (Tx-Rx)  and  reflector  (Rr).  The  shape  of  the  moving  object  is  thus  traced  at 
the  output  of  the  sensor  system.  The  choice  of  number  of  sensors  and  their  positions  on 
the  poles  are  important  design  parameters.  This  effort  investigates  techniques  to  optimize 
these  parameters  with  the  objective  of  enhancing  the  quality  of  shape  features  that 
distinguish  humans  from  animals.  Each  Tx-Rx  and  Rr  pair  generates  ‘O’  based  on  the 
amount  of  time  they  have  been  blocked  by  the  moving  object.  The  count  of  the  number  of 
‘0’s  generated  by  a  pair  can  be  used  one  feature.  The  sensor  system  used  in  this  research 
has  16  such  pairs,  generating  16  features  for  each  object  (hence  forth  a  Tx-Rx  and  Rr  pair 
will  be  referred  to  as  a  detector  in  this  report).  This  means  a  feature  fusion  technique  that 
assigns  weights  to  features  based  on  their  importance  also  assigns  weights  to  the 
corresponding  detector.  This  means  that  a  detector  with  negligible  weights  can  be 
removed  from  the  sensor  during  subsequent  design  iterations.  Three  feature  fusion 
techniques,  namely  Lasso  [Tibshiranil996],  Group  Lasso[Friedman2010]  and  Sparse 


Multiple  Kernel  Leaming[Subramanhya2010]  are  investigated  for  feature  quality  based 
sensor  design. 


Sparse  Multiple  Kernel  Learning  (SMKL) 

In  many  cases,  two  classes,  not  separable  linearly  in  their  native  dimensions,  can  be  made 
linearly  separable  in  higher  dimensions.  In  algorithms  such  as  Support  Vector  Machines 
(SVM),  the  inner  product  operation  on  the  data  is  an  important  step.  This  process 
becomes  computationally  challenging  in  higher  dimensions.  The  Kernel  trick  allows  the 
dot  product  to  be  implemented  in  the  lower  dimensions  itself,  but  gives  the  effect  of  a  dot 
product  in  the  higher  dimension.  Let  (p{x )  represent  the  mapping  or  projection  of  x  on  to 

a  higher  dimension.  The  Kernel  trick  allows  for  the  inner  product  in  the 

higher  dimensional  feature  space  to  be  represented  by  a  Kernel  in  the  input  (low 
dimension)  space  K(x, x')  .  For  example  the  radial  basis  function  Kernel  given  by 
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K(x,x')  =  exp  • 
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corresponds  to  an  inner  product  in  an  infinite  dimensional 


space. 


SMKL  is  a  non-linear  group  selection  technique  that  weights  groups  of  features.  In  the 
context  of  this  report,  the  technique  assigns  weights  to  groups  of  detectors.  SMKL 
consists  of  a  composite  K(x,x’)  kernel  generated  by  the  weighted  sum  of  primitive  kernels 
k(x,x  ■). 


K(x,x')  =  ^pgk(x,x') 
<■*1 


where  G  is  the  total  number  of  group  and  f$s  are  the  kernel  weights.  If  different  groups  of 
features  are  assignment  to  different  kernels  then  the  kernel  weights  become  weights  for 
the  groups.  The  process  of  estimation  of  the  weights  follows  the  technique  described  in 
[Subramanhya2010] 


Lasso 

Least  Absolute  Selection  and  Shrinkage  Operator,  known  more  by  its  acronym  Lasso  is  a 
linear  technique  for  solving  a  sparse  regression  problem.  Lasso  minimizes  objective 

function  || X/?-  y\f2  +  A|| /J||  to  find  the  optimum  weighting  parameters  /?  for  the  features  X 

given  the  corresponding  class  predictions  y.  X  is  the  regularization  parameter  that  can  be 
varied  to  change  the  sparsity  of  the  solution. 


Group  Lasso 


Group  Lasso  as  proposed  by  Friedman  et  al.  is  regression  technique  that  introduces 
sparsity  at  both  the  group  level  and  at  the  individual  feature  level  The  sparsity  is 
achieved  by  minimizing  the  cost  function  given  by: 


■42W  +*>14 


G  is  the  number  of  groups,  Ng  is  the  number  of  features  in  the  gth  group,  Xg  corresponds 

to  the  features  of  the  gth  group,  A  =  [AA  -*A?]  A  =[A A— Process  °f 

solving  the  optimization  problem  is  described  in  [Friedman2010]  The  second  term  in  the 
cost  function  can  cause  very  few  weights  in  ft q  to  be  non-zero  introducing  sparsity  within 
a  group.  A  combination  of  the  second  and  third  term  in  the  cost  function  can  cause  all  the 
weight  in  a  particular  to  be  set  to  zero.  This  results  in  discarding  an  entire  group, 
leading  to  sparsity  at  the  group  level.  In  summary,  Group  Lasso  provides  both  inter-group 
and  intra-group  sparsity,  It  should  be  noted  that  the  SMKJL  technique  only  provides  group 
level  sparsity  and  not  intra-group  sparsity. 


Results 

The  algorithms  were  applied  to  a  version  of  the  trip 
wire  sensor,  which  has  16  detectors.  In  the  case  of 
Group  Lasso,  three  groups  were  formed  with 
detectors  1  through  5  forming  group  1,  detectors  6 
to  11  forming  group  2  and  detectors  12  through  16 
forming  group  3.  A  total  of  135  human  profiles  and 
96  animal  profiles  collected  through  various  data 
collections  were  used  for  the  analysis.  Figure  3.1 .  Weights  assigned  to 

detectors  through  Lasso 

The  graphs  shown  in  Figures  3.1  and  3.2  show  the  weights  assigned  to  detectors  of  the 
trip  wire  sensor  using  Lasso  and  Group  Lasso  respectively. 


The  results  indicate  Lasso  and  Group  Lasso  assigns 
non-zero  weights  to  only  7  out  of  the  total  of  16 
detectors.  In  this  particular  case,  it  can  be 
hypothesized  that  Group  Lasso  does  not  provide 
group  level  sparsity  since  at  least  one  detector  in 
each  group  is  essential  to  minimizing  the  first  term 
in  the  cost  function.  SMKL  provides  weights  of 
0.3804,  0.305  and  0.3.144  respectively  for  groups  1, 
2  and  3.  This  confirms  with  the  results  of  Lasso  and 
Group  Lasso,  since  every  group  has  detectors  that 
contribute  toward  minimizing  the  error. 


Figure  3.2.  Weights  assigned  to 
detectors  using  group  Lasso 


The  investigation  indicates  that  the  Group  Lasso  technique  is  an  effective  for  feature 
quality  maximizing  sensor  design  because  of  its  ability  to  provide  both  inter-group  and 
intra-group  feature  sparsity.  Future  effort  will  entail  research  into  muth-modal  sensor 
network  design  using  the  techniques  described  above 
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4.0  Sensor  Ontologies 

Task  4.1 :  Conduct  basic  research  to  a)  advance  sensor  ontologies  for 
problem-solving  architecture;  and  b)  advance  profiling  sensors  through  the 
use  of  innovative  classification  algorithms  and  data-visualization 
techniques 

The  deployment  of  ubiquitous  sensor  systems  and  algorithms  has  led  to  many  challenges, 
such  as  matching  sensor  systems  to  compatible  algorithms  which  are  capable  of 
satisfying  a  task.  Compounding  the  challenges  is  the  lack  of  the  requisite  knowledge 
models  needed  to  discover  sensors  and  algorithms  and  to  subsequently  integrate  their 
capabilities  to  satisfy  a  specific  task.  Basic  principles  for  an  ontological  problem-solving 
framework  have  been  researched  and  a  proof-of-concept  framework  was  implemented  in 
support  of  Task  4.1. 

The  proof-of-concept  framework  matches  sensors  to  compatible  algorithms  to  form 
synthesized  systems,  which  are  capable  of  satisfying  a  task  and  then  assigning  the 
synthesized  systems  to  high-level  missions.  The  approach  designed  for  the  ontological 
problem-solving  framework  has  been  instantiated  in  the  context  of  a  persistence 
surveillance  prototype  environment,  which  includes  profiling  sensor  systems  and 
algorithms  to  demonstrate  proof-of-concept  principles.  Even  though  the  problem-solving 
approach  was  instantiated  with  profiling  sensor  systems  and  algorithms,  the  ontological 
framework  may  be  useful  with  other  heterogeneous  sensing-system  environments.  Work 
in  support  of  Task  4.1  was  published  in  J.  Qualls  and  D.J.  Russomanno  (2011) 
“Ontological  Problem-Solving  Framework  for  Dynamically  Configuring  Sensor  Systems 


and  Algorithms,”  Sensors ,  Volume  11,  Number  3,  3177-3204.  This  paper  is  provided  as  a 
supplement  to  this  report. 

The  work  of  the  previous  paragraph  was  extended,  also  in  support  of  Task  4.1.  This  work 
leverages  knowledge  models  describing  sensors,  algorithms,  and  high-level  missions  to 
facilitate  automated  inference  of  assigning  systems  to  subtasks  that  may  satisfy  a  given 
mission  specification.  To  demonstrate  the  efficacy  of  the  ontological  problem-solving 
architecture,  a  family  of  persistence  surveillance  sensor  systems  and  algorithms  has  been 
instantiated  in  a  prototype  environment  to  demonstrate  the  assignment  of  systems  to 
subtasks  of  high-level  missions.  This  work  extends  the  results  of  the  previously  cited 
journal  paper  and  was  published  as  J.  Qualls  and  D.J.  Russomanno  (2011)  “Ontological 
Problem-Solving  Framework  for  Assigning  Sensor  Systems  and  Algorithms  to  High- 
Level  Missions”  Sensors ,  Volume  11,  Number  9,  8370-8394.  This  paper  is  provided  as  a 
supplement  to  this  report. 

To  advance  profiling  sensors  through  the  use  of  innovative  data-visualization  techniques 
work  was  completed  in  support  of  Task  4.1  to  port  existing  data  visualization  software 
for  creating  silhouettes  from  data  obtained  from  the  vertical  array  detector  version  of  a 
profiling  sensor  to  a  platform-independent  architecture.  The  silhouette  viewer  was 
subsequently  integrated  into  the  ontology  problem-solving  framework  where  it  can  be 
described  using  the  web  ontology  language  (OWL)  and  subsequently  discovered  and 
tasked.  The  integration  of  the  silhouette  viewer  into  the  ontology-based  framework  is 
described  in  the  context  of  data  fusion  in  the  paper  C.  Kothari,  J.  Qualls  and  D.J. 
Russomanno  (2012)  “An  Ontology-Based  Data  Fusion  Framework  for  Profiling 
Sensors,”  IEEE  International  Conference  on  Electro/Information  Technology ,  IEEE 
Press,  Indianapolis,  Indiana.  The  research  in  this  paper  was  also  in  direct  support  of  Task 
1  and  is  provided  as  a  supplement  to  this  report. 

Additional  work  was  completed  in  support  of  Task  4.1  in  classification  algorithm 
development.  A  back-propagation  neural  network  was  created  to  classify  data  from  a 
novel,  wireless  profiling  sensor  developed  in  support  of  Task  4.3.  The  wireless  version  of 
the  profiling  sensor  allows  the  detectors  to  be  placed  in  custom  configurations,  versus  the 
traditional  vertical  detector  array.  The  results  of  the  classification  algorithm  were 
published  in  the  paper.  A.  Galvis,  D.J.  Russomanno  and  C.  Kothari  (2012)  “A  Wireless 
Near-IR  Retro-Reflective  Profiling  Sensor Proceedings  SPIE:  Ground/ Air  Multi-Sensor 
Interoperability ,  Integration  and  Networking  for  Persistent  ISR  Ilf  Volume  8389, 
Baltimore,  Maryland.  This  paper  is  also  provided  as  a  supplement  to  this  report. 


Task  4.2:  Conduct  research  to  help  define  a  means  by  which  sensor 
specifications,  capabilities,  and  properties  can  be  published  and 
discovered  in  a  computer-readable  format 

The  ontology  work  described  in  support  of  Task  4.1  also  directly  supports  Task  4.2  by 
using  ontologies  to  define  and  make  available  in  a  computer-readable  fonnat  sensor 
specifications,  capabilities,  and  properties.  In  addition  to  the  two  Sensors  journal  papers 


previously  listed,  additional  research  was  conducted  in  support  of  Task  2  to  provide  a 
proof-of-concept  illustration  about  how  sensor  specifications,  capabilities,  and  properties 
can  be  published  in  an  ontology  and  used  to  support  ontology-based  fusion.  The  paper 
published  as  C.  Kothari,  J.  Qualls  and  D.J.  Russomanno  (2012)  “An  Ontology -Based 
Data  Fusion  Framework  for  Profiling  Sensors,”  IEEE  International  Conference  on 
Electi-o/Information  Technology,  IEEE  Press,  Indianapolis,  Indiana,  which  was  first  cited 
above  in  support  of  the  visualization  subtasks  of  Task  4.1,  describes  the  details  of  this 
work  in  support  of  Task  4.2. 

Additional  activity  and  accomplishments  in  support  of  Task  4.2  include  a  concept  paper 
researching  the  role  of  an  ontology  to  support  data-to-decision  sensing  environments  that 
assess  human  intent  from  external  stimuli.  The  paper  published  as  C.  Kothari,  D.J 
Russomanno,  R.B.  Sartain  and  R.  Frankel  (2012)  “Toward  Data-to-Decision  Sensing 
Environments  to  Assess  Human  Intent  from  Responses  to  Stimuli,”  Proceedings  SPIE: 
Ground' Air  Multi-Sensor  Interoperability,  Integration  and  Networking  for  Persistent  1SR 
III,  Volume  8389,  Baltimore,  Maryland,  is  also  provided  as  a  supplement  to  this  report. 


Task  4.3:  Assess  research  theories  and  concepts  through  the  construction 
of  laboratory  sensors,  algorithm  development  and  implementation, 
implementation  of  novel  visualization  software,  and  hardware  and  software 
architectural  considerations  that  could  ultimately  be  integrated  into 
network-centric  architecture 

Profiling  sensors  were  advanced  in  support  of  Task  4.3  through  the  construction  of  novel 
laboratory  sensors,  algorithms,  and  visualization  tools.  Three  laboratory  profiling  sensors 
were  created  in  support  of  Task  4.3,  including  a  traditional  vertical  array  (and  wired) 
profiling  sensor,  a  wired  profiling  sensor  with  offset  detectors,  and  a  novel,  wireless 
profiling  sensor  in  which  each  detector  comprises  a  node  in  a  wireless  sensor  network. 
The  wireless  sensor  network  approach  allows  the  detectors  of  a  profiling  sensor  to  be 
deployed  in  custom  configurations.  Software  to  classify  and  visualize  data  acquired  from 
the  novel,  wireless  profiling  sensor  was  also  developed  in  support  of  Task  4.3. 

The  notion  of  a  profiling  sensor  was  first  implemented  as  a  near-IR,  retro-reflective 
prototype  consisting  of  a  vertical  column  of  sparse  detectors  by  a  team  led  by  Dr. 
Russomanno  at  the  U.  of  Memphis  in  support  of  cooperative  agreement  W911NF-05-2- 
0019  between  the  University  of  Memphis  and  the  U.S.  Army’s  Research  Laboratory 
(ARL).  Alternative  arrangements  of  detectors  were  researched  and  implemented  in 
support  of  Task  4.3  in  which  a  subset  of  the  detectors  were  offset  from  the  vertical 
column  and  placed  at  arbitrary  locations  along  the  anticipated  path  of  the  objects  of 
interest.  The  paper  published  by  R.K.  Reynolds,  S.  Chari  and  D.J.  Russomanno  (2011) 
“Embedded  Real-Time  Classifier  for  Profiling  Sensors  and  Custom  Detector 
Configuration,”  Proceedings  SPIE:  Ground/Air  Multi-Sensor  Interoperability , 
Integration  and  Networking  for  Persistent  ISR ,  Volume  8047,  Orlando,  Florida,  pp. 
80470E-1-80470E-9,  is  also  provided  as  a  supplement  to  this  report  and  describes 
accomplishments  in  support  of  Task  4.3. 

All  prior  work  with  the  near-IR,  retro-reflective  profiling  sensors  has  consisted  of  wired 
detectors,  including  the  work  cited  in  the  previous  paragraph.  Research  was  conducted  in 
support  of  Task  4.3  to  advance  this  prior  work  by  designing  and  implementing  a  wireless 
prototype  version  of  a  near-IR,  retro-reflective  profiling  sensor  in  which  each  detector  is 
a  wireless  sensor  node.  In  this  architecture,  a  base  station  is  responsible  for  collecting  all 
data  from  the  detector  sensor  nodes  and  coordinating  all  pre-processing  of  data  collected 
from  the  sensor  nodes,  including  data  re-alignment,  before  subsequent  classification 
algorithms  are  executed.  Such  a  wireless  detector  configuration  advances  deployment 
options  for  near-IR,  retro-reflective  profiling  sensors.  This  work,  which  was  in  support  of 
Task  4.3,  is  detailed  in  the  paper:  A.  Galvis,  D.J.  Russomanno  and  C.  Kothari  (2012)  “A 
Wireless  Near-IR  Retro-Reflective  Profiling  Sensor,”  Proceedings  SPIE:  Ground/Air 
Multi-Sensor  Interoperability,  Integration  and  Networking  for  Persistent  ISR  III,  Volume 
8389,  Baltimore,  Maryland.  This  paper  is  also  provided  as  a  supplement  to  this  report. 


Task  4.4:  Undertake  field  data  collection  with  profiling  sensors  and 
emulators  to  build  an  extensive  signature  library  to  support  algorithm 
development 

In  support  of  Task  4.4,  field  data  collections  were  limited  to  expanding  the  profiling 
sensor  library  with  data  captured  from  the  new  profiling  sensors  that  were  developed  in 
support  of  Task  4.3.  The  data  was  used  to  support  the  classification  algorithm  and 
visualization  algorithm  development  referenced  in  support  of  Tasks  4.1  through  4.3. 
Additional  field  data  collections  would  improve  the  robustness  of  the  various  classifiers. 
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5.0  Profiling  Sensors:  classifiers,  visualization 
techniques,  and  network-centric  integration 


Background 

Conventional  electro-optical  systems  have  been  used  extensively  in  Intelligence 
Surveillance  and  Reconnaissance  (ISR)  applications.  These  systems  have  typically  been 


based  on  using  two  dimensional  focal  plane  arrays.  The  disadvantage  of  such  systems  for 
applications  of  interest,  namely  deployment  in  inaccessible  terrains  for  border  security,  is 
power  consumption  and  cost.  If  the  discrimination  task  is  narrowed  down  to  specific 
classes  such  as  distinguishing  between  humans,  animals  and  vehicles  at  ranges  not 
exceeding  few  tens  of  meters,  high  resolution,  high  bit  depth  imaging  systems  may  not  be 
required.  This  scenario  is  true  especially  in  the  terrains  in  which  objects  can  only  travel 
through  specific  routes  consisting  of  narrow  trails.  In  this  research,  object  recognition 
algorithms  are  developed  for  a  pyroelectric  detector  based  linear  array  to  classify  humans 
and  animals.  Initial  testing  on  a  smaller  set  of  data  point  showed  promising  results 
[White2010]  .  The  current  report  presents  findings  based  on  a  more  extensive  data 
collection  effort  and  new  algorithmic  approaches. 


Sensor  Description 


A  schematic  of  the  pyroelectnc  linear  array  (PLA)  sensor  is  shown  in  Figure  5.1  and  a 
photo  of  the  sensor  package  is  shown  in  Figure  5.2.  The  PLA  sensor  uses  a  Dias  128 
element  linear  array  of  pyroelectric  detectors  along  with  a  F/0.86  germanium  lens.  The 
size  of  each  detector  is  90|im  x  lOOpm  with  a  pitch  of  lOpm.  With  a  focal  length  of  50 
mm,  the  detector  instantaneous  FOV  (IFOV)  is  1.8mrads  x  2mrads.  The  extent  of  the 
spatial  sample  at  range  of  30  meters  is  5.4cm  x  6cm,  providing  about  17  samples  over  the 
height  of  a  2m  tall  human.  A  18F4550  pic  microcontroller  is  used  for  A/D  conversion  and 
communication.  The  sensor  system  operates  at  a  sampling  rate  of  20Hz. 


Figure  5.1.  Schematic  of  PLA  sensor. 


Figure  5.2.  Photo  of  PLA  sensor 


Data  collection  effort 


Data  collection  using  the  PLA  sensor  was  performed  in  two  geographically  distinct 
locations.  One  location  was  near  the  US-Mexico  border  which  is  typically  has  an  arid 
terrain  with  thorn  bushes  forming  significant  portions  of  the  vegetation.  The  other 
location  was  at  a  petting  zoo  near  Memphis,  Tennessee  where  the  terrain  was  covered 
with  grass  and  trees.  The  human  category  was  represented  by  males  and  females  of  with 
varying  physical  built.  Large,  medium  and  small  horses,  cows,  lamas,  donkeys  and  dogs 
formed  the  animal  class.  The  humans  and  animals  moved  in  the  field  of  view  of  the 
system  at  various  speeds  at  ranges  varying  from  10  meters  to  20  meters.  Animals  had 
handlers  directing  their  movements  during  the  data  collection.  Figure  5.3  shows  the 
output  of  the  PLA  sensor  with  three  horses,  Figure  5.4  shows  six  humans  and  Figure  5.5 
shows  a  human  followed  by  a  miniature  cow. 


Figure  1.3.  Image  of  three  horses  generated  by  the  PLA  sensor 


Figure  5.4.  Image  of  six  humans  generated  by  PLA  sensor 


Figure  5.5.  Image  of  a  human  followed  by  a  miniature  cow  generated  by  PLA  sensor 


Object  Recognition  algorithms 

A  statistical  threshold  was  estimated  for  each  detector  output  for  segmenting  the  targets 
from  background.  The  AC  couple  nature  of  the  pyroelectric  detector  suppresses  static 
objects  in  the  background  and  only  responds  to  the  moving  targets.  If  a  target  does  not 
show  variations  in  sections  of  its  body,  then  the  detectors  do  not  respond.  This  can  lead  to 
fragmentation  of  an  object.  When  a  hot  object  enters  and  leaves  the  FOV  of  the  sensor, 
the  step  response  properties  of  the  pyroelectric  detector  generate  large  output  up  swings 
and  down  swings  at  the  instants  of  entry  and  exit  of  the  object  respectively.  Fragmented 
objects  can  be  regrouped  by  tracking  these  bipolar  swings.  The  statistical  threshold  used 
for  segmentation  was  also  used  to  binarize  objects  with  object  set  to  1  and  background  set 
to  0  for  extracting  geometrical  features  such  as  height  to  width.  On  the  other  hand,  the 
segmented  objects  were  retained  in  gray  scale  form  for  extracting  texture  based  features 
using  Log-Gabor  filter  bank.  These  features  are  then  classified  using  one  of  three 
different  classification  algorithms,  namely  Mahalanobis  distance  classifier,  Gaussian 
mixture  classifier  and  Logistic  Regression. 


Results 

A  total  of  315  human  profiles  and  182  animal  signatures  were  collected  using  the 
pyroelectric  sensor  at  two  locations  at  the  US  Mexico  border  in  Arizona  (ArzData)  and  at 
a  petting  zoo  near  Memphis  (PzData).  The  number  of  samples  of  at  each  location  for  the 
three  data  collection  sites  is  shown  in  Table  5.1.  The  data  was  separated  into  testing  and 
training  data  using  in  two  ways  as  shown  in  Table  5.2.  The  training  and  testing  choices 
described  in  Table  5.2  were  made  because  PzData  has  too  few  samples  to  form  a  training 
set  in  itself.  On  the  other  hand,  including  PzData  in  the  test  set  increased  the  variance  of 
the  test  set,  making  the  test  robust.  This  is  because  the  animal  samples  in  ArzData  were 
horses  but  the  petting  zoo  data,  apart  from  horses,  also  contained  cows,  lamas,  donkeys. 


Table  5.1:  Number  of  data  points  collected  using  PLA  sensor 


ArzData  location  1 

ArzData  location  2 

PzData 

human  signatures 

140 

145 

41 

animal  signatures 

73 

84 

41 

Table  5.2:  Training  and  testing  data 


Data  set 

Training  data 

Test  data 

1 

ArzData  location  1 

ArzData  location  2  + 

PzData 

2 

ArzData  location  2 

ArzData  location  1  + 

PzData 


Classification  rates  are  shown  in  Tables  5.3  and  5.4.  It  is  observed  that  Logistic 
regression  using  height  and  width  features  achieves  the  highest  classification  rate  of  over 
89%  for  data  set  1  and  classification  rate  of  over  90%  classification  rate  for  data  set  2. 
The  GMM  classifier  using  height  to  width  ratio  also  performs  well  coming  with 
classification  rates  over  85%  in  both  cases.  The  main  cause  of  error  was  the  incorrect 
grouping  of  fragments  of  objects  during  the  object  detection  phase.  Another  cause  of 
error  was  animals  running  at  high  speeds.  The  profile  height  and  width  feature  of  a 
galloping  horse  is  similar  to  that  of  a  tall  human  walking  slowly.  The  Gabor  features 
performed  poorly  and  using  dimensionality  reduction  techniques  such  as  linear 
discriminant  analysis  only  provided  marginal  improvement  in  the  classification  accuracy. 
Though  results  of  the  classification  rates  are  not  reported  here  for  wavelet  packet  based 
features,  their  classification  performance  was  significantly  lower. 


Table  5.3.  Classification  rates  for  dataset  1 


Feature  extraction 

Classifier 

Classification  rate 

Gabor  features 

Mahalanobis  distance 

75.3% 

Height  and  width 

Logistic  Regression 

89.6% 

Height  to  width  ratio 

GMM 

85.7% 

Table  5.4.  Classification  rates  for  data  set  2 


Feature  extraction 

Classifier 

Classification  rate 

Gabor  features 

Mahalanobis  distance 

73.6% 

Height  and  width 

Logistic  Regression 

90.8% 

Height  to  width  ratio 

GMM 

86.4% 

Future  Work 

The  results  from  this  research  effort  will  be  submitted  to  the  IEEE  Sensors  Journal  in 
August  2011. 

Several  tasks  are  foreseen  in  future  efforts  to  further  refine  and  improve  the  sensor 
system.  The  first  one  includes  implementing  the  software  on  low  resource  computing 
platforms  for  real  time  performance.  The  second  task  is  incorporating  this  system  into  a 
multi-modal  sensor  network.  It  is  hypothesized  that  fusion  of  features  from  the  PLA 
sensor  with  sensors  such  as  acoustic  and  seismic  sensors  will  further  increase  the 
discrimination  rates.  Another  goal  is  to  overcome  misclassification  of  galloping  animals 
as  humans.  For  this  a  two  columns  array  is  currently  being  developed.  The  time  of 


appearance  of  an  object  in  each  of  the  columns  can  be  used  to  estimate  speed  of  objects, 
which  can  be  used  to  address  the  above-mentioned  errors.  Further,  a  more  sensitive 
detector  array  is  being  used  for  the  development  of  a  long  range  PLA  with  ability  to 
discriminate  well  beyond  the  30-meter  range  limit  of  the  current  PLA  system.  This  long 
range  PLA  system  will  provide  greater  sensor  to  object  standoff  distance  and  thus 
expanding  the  utility  of  the  sensor  beyond  trail  monitoring  into  other  tactical  and  military 
applications.  The  higher  sensitivity  detector  system  can  reduce  the  fragmentation  effect  at 
the  sensor  output  that  has  also  been  a  source  of  classification  errors. 
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6.0  Alternative  Sensing  Modalities 

Task  6.1:  Develop  alternate  modalities  of  operation  for  beam  break  profiling 
sensors. 

An  alternative  to  beam-break  sensors  for  border  and  perimeter  security  is  the  use  of 
micro-mechanical  resonators.  These  types  of  sensors  could  be  used  to  produce  seismic  or 
acoustic  versions  of  profiling  sensors.  A  detailed  analysis  of  this  mode  of  sensing  was 
conducted  and  is  presented  in  Appendix  X. 

Task  6.2:  Map  the  feature  space  of  an  “atomic”  sensor  and  relate  the 
features  to  profile  classification  performance. 

Sensor  atoms  are  fundamental  units  of  sensing.  This  concept  was  explored  as  part  of  the 
overall  effort  of  this  contract.  Some  basic  properties  of  a  particular  type  of  sensor  atom, 
the  Bernoulli  sensor  atom,  were  derived.  A  brief  paper  (unpublished)  on  the  concept  of 
sensor  atoms  and  the  Bernoulli  sensor  atom  was  produced  and  is  provided  in  Appendix 
X.  Publication  of  this  work  will  require  implementation  and  refinement  of  the  concept  of 
atoms  in  groups,  which  is  an  ongoing  area  of  research. 


Task  6.3:  Further  research  into  a  University  of  Memphis  concept  for 
terahertz  and  sub-millimeter  wave  imaging  for  concealed 
weapons/contraband  detection. 

Task  6.4:  Investigate  the  use  of  compressive  sensing  techniques  in 
improving  profile  classification. 

THz/sub-millimeter  wave  sensing  provides  a  “see  through”  capability  without  the  use  of 
ionizing  radiation.  This  capability  is  important  in  many  types  of  security  problems  where 
the  potential  threat  posed  by  an  individual  must  be  assessed  prior  to  entry  into  a  secured 
area  Sensors  of  this  type  have  been  under  development  for  some  time  and  low  cost 
solutions  remain  elusive.  Several  sub-millimeter  wave  active  and  passive  imaging 
systems  for  detection  of  hidden  threats  have  been  demonstrated  but  they  all  suffer  from 
the  low  frame  rate  (i.e.  active  imagers)  or  high  size  and  power  consumption  associated 
with  cooled  detection  (i.e.  passive  imagers).  Our  effort  is  focused  toward  a  solution  to 
these  problems  using  image  plane  coded  aperture  imaging  techniques  in  active 
configurations  with  single  pixel  detectors.  Different  aspects  of  the  work  have  been 
published  in  peer  reviewed  conferences  and  journals.  Invited  talks  on  this  approach  have 
been  presented  at  IEEE  and  OSA  conferences. 

At  the  beginning  of  the  effort  sub-millimeter  wave  imaging  using  a  single  pixel  detector 
and  an  image  plane  coded  aperture  had  been  demonstrated  by  our  group.  The  image  plane 
coded  aperture  was  implemented  using  a  spinning  disk  with  holes  separated  in  a  pseudo¬ 
random  fashion  on  a  perimeter  line.  The  proof  of  concept  device  could  reconstruct  line 
images  of  a  masked  source  placed  ten  meters  away  from  the  aperture  of  a  reflective  optic. 
These  simple  images  consisted  of  two  point  sources.  The  work  was  published  in  the 
paper  0.  Furxhi  and  E.  L.  Jacobs  (2010)  “A  Sub-Millimeter  Wave  Line  Imaging  Device,” 
Proceedings  of  SPIE,  Volume  7670,  Orlando,  Florida,  USA.  This  paper  is  provided  as  a 
supplement  to  this  report. 

In  continuation  of  the  effort  described  in  the  previous  paragraph,  a  scanning  line  imager 
was  built  and  imaging  of  an  extended  metallic  object  was  demonstrated.  Each  line  of  the 
image  was  acquired  in  300ms  as  the  target  was  scanned  across  the  field  of  view.  The 
images  were  reconstructed  offline  using  linear  measurement  techniques.  The  linear 
measurement  techniques  are  computationally  efficient  and  the  image  reconstruction 
reduces  to  a  simple  matrix  vector  multiplications.  This  work  was  published  in  the  paper 
O.  Furxhi  and  E.  L.  Jacobs  (2010)  “A  sub-millimeter  wave  line  scanning  imager,” 
Proceedings  of  SPIE,  Volume  7837,  Toulouse,  France.  This  paper  is  provided  as  a 
supplement  to  this  report. 

The  image  plane  coded  aperture  used  in  the  previous  work  is  suitable  for  building  a 
compressive  sensing  (CS)  imager.  CS  imagers  make  linear  measurements  on  the  image  of 
the  scene.  The  number  of  measurements  is  usually  smaller  than  the  desired  number  of 
pixels  in  the  image  and  therefore  the  image  reconstruction  is  realized  using  non-linear 
iterative  algorithms.  We  used  the  CS  algorithms  on  measurements  collected  with  the 
image  plane  coded  aperture  line  imager  and  were  able  to  reconstruct  line  images  with  half 
the  number  of  measurements  used  with  linear  measurement  techniques.  The  line  imager 


configuration  of  this  sensor  is  in  essence  a  THz  profiling  sensor.  This  work  was 
summarized  in  the  paper  Imama  Noor,  Orges  Furxhi  and  Eddie  L.  Jacobs  (2011) 
''Compressive  sensing  for  a  sub-millimeter-wave  single  pixel  imager",  Proceedings  of 
SPIE,  Volume  8022,  Orlando,  Florida,  USA.  This  paper  is  provided  as  a  supplement  to 
this  report. 

In  continuation  of  the  effort,  a  two-dimensional  version  of  the  image  plane  coded 
aperture  imager  was  built  and  demonstrated.  The  image  plane  coded  aperture  was 
implemented  using  a  spinning  disk  with  holes  placed  pseudo  randomly  across  the  surface 
of  the  disk.  A  square  aperture  was  placed  in  front  of  the  disk  to  indicate  an  image 
window.  This  device  could  scan  a  two-dimensional  image  of  the  scene  every  20ms 
(50Hz).  The  images  that  were  reconstructed  consisted  of  64x64  pixels  and  were 
reconstructed  in  real  time  as  the  data  was  collected.  Initially  the  two-dimensional  coded 
aperture  was  used  to  fonn  images  at  visible  light  wave  lengths.  For  this  purpose  an  N- 
type  silicon  PIN  photodetector  was  used  instead  of  the  sub-millimeter  wave  detector.  And 
a  collimated  red  laser  was  used  as  an  illumination  source  instead  of  the  sub-millimeter 
wave  source.  The  reason  for  visible  light  implementation  was  two-fold.  First,  it  helped 
with  the  debugging  of  the  issues  related  to  the  spinning  disk  coded  aperture  (elimination 
issues  with  the  sub-millimeter  wave  optics)  and  second,  it  demonstrated  the  wavelength 
independence  of  the  spinning  disk  coded  aperture.  This  work  was  published  in  the  paper 
O.  Furxhi  and  E.  Jacobs,  "Spatially  Selective  Mask  for  Single  Pixel  Video  Rate 
Imaging,"  in  Imaging  Systems  and  Applications,  OSA  Technical  Digest  (CD)  (Optical 
Society  of  America,  2011),  paper  ITuA3,  in  Toronto,  Ontario,  Canada.  This  paper  is 
provided  as  a  supplement  to  this  report. 

The  spinning  disk  coded  aperture  device  was  then  used  to  implement  a  two-dimensional, 
real-time,  sub-millimeter  wave  imager.  This  device  could  scan  a  two-dimensional 
reflection  mode  image  of  a  metallic  target  positioned  10  meters  away,  eveiy  20ms 
(50Hz).  The  target  was  illuminated  with  sub-millimeter  waves  and  its  reflection  was 
imaged  on  the  coded  aperture.  The  images  that  were  reconstructed  consisted  of  64x64 
pixels  and  were  reconstructed  in  real  time  as  the  data  was  collected.  Imaging  was  also 
demonstrated  in  transmission  mode  (imaging  the  transmission  of  a  sample/target).  This 
work  was  published  in  Orges  Furxhi,  Eddie  L.  Jacobs  and  Robert  Hewitt  (2011),  "Two- 
dimensional,  real-time,  sub-millimeter-wave  imaging  using  a  spatially  selective  mask", 
Proceedings  of  SPIE,  Volume  8022,  Orlando,  Florida,  USA.  This  paper  is  provided  as  a 
supplement  to  this  report. 

An  analysis  comparing  various  modes  of  active  imaging  in  terms  of  signal  to  noise  ratio 
(SNR),  including  the  image  plane  coded  aperture  approach,  was  conducted.  The  purpose 
of  the  analysis  was  the  identification  of  advantages  and  disadvantages  of  the  image  plane 
coded  aperture  approach  compared  to  more  traditional  ways  of  forming  images  in  active 
configurations.  The  analysis  concluded  that  the  measurement  SNR  performance  of  the 
image  plane  coded  aperture  approach  rests  between  the  conjugate  point  imager 
configurations  (best  SNR)  and  focal  plane  array  configurations  (worst  SNR)  provided 
equal  illumination  power.  The  SNR  approached  the  best  SNR  case  when  compressive 
sensing  reconstruction  techniques  could  be  realized.  However,  the  image  SNR  for  the 


coded  aperture  approach  is  also  related  to  the  reconstruction  method.  The  analysis 
identified  the  need  for  further  analysis  with  regards  to  methods  for  the  generation  of  hole 
patterns  on  the  spinning  disk  that  lead  to  independent  codes  (independent  codes  lead  to 
better  reconstructions).  Another  approach  that  could  be  pursued  is  the  identification  of 
other  implementation  of  the  coded  aperture  different  from  the  spinning  disk.  This  work 
was  published  in  Orges  Furxhi  and  Eddie  L.  Jacobs  (2011),  "Comparison  of  schemes  for 
active  sub-millimeter  wave  imaging",  Proceedings  of  SPIE,  Volume  8188,  Prague,  Czech 
Republic.  This  paper  is  provided  as  a  supplement  to  this  report. 

Further  analysis  concerning  the  image  reconstruction  technique  that  is  utilized  by  the  two 
dimensional  sub-millimeter  wave  imager  was  conducted  and  a  journal  paper  was 
published  in  the  Journal  of  Optical  Engineering  summarizing  the  imager  and  the 
mathematical  analysis  associated  with  the  reconstruction  techniques.  The  publication 
Orges  Furxhi,  Eddie  L.  Jacobs  and  Chrysanthe  Preza,  "Image  plane  coded  aperture  for 
terahertz  imaging".  Opt.  Eng.  51,  091612  (Jun  15,  2012),  is  provided  as  a  supplement  to 
this  report. 

Other  approaches  of  implementing  an  image  plane  coded  aperture  imager  were 
investigated  and  an  effort  was  put  forward  to  formalize  the  concept  of  image  plane  coded 
aperture  (IPCA)  detectors.  IPCA  detectors  are  used  as  substitutes  for  focal  plane  arrays  in 
frequency  regimes  where  focal  plane  arrays  are  impractical,  expensive,  or  non-existent. 
IPCA  detectors  are  composed  of  a  single  element  detector  sensitive  to  the  radiation 
frequency  of  interest,  a  reconfigurable  spatial  light  modulator  (SLM),  and  a  mechanism 
such  as  a  lens  or  a  hom  that  is  used  to  collect  the  radiation  past  the  SLM  and  focus  it  on 
the  sensing  element.  The  IPCA  detector  is  placed  in  the  image  plane  of  an  imager  and  is 
used  to  make  linear  measurements  on  the  image  by  modulating  the  information  on  the 
image  plane  spatially  and/or  temporally  in  amplitude,  frequency,  phase,  or  polarization. 
The  image  is  then  reconstructed  computationally  using  inverse  imaging  techniques.  In 
this  work  the  spinning  disk  IPCA  detector  for  sub-millimeter  waves  was  used  to  illustrate 
the  concept.  Also,  an  IPCA  detector  with  a  phase  modulating  SLM  was  proposed  as  an 
alternative.  This  work  was  published  in  O.  Furxhi,  E.  L.  Jacobs  (2012),  “Image  Plane 
Coded  Aperture  Detectors  for  THz  Imaging”  2012  IEEE  MTT-S  International 
Microwave  Symposium  Digest,  Montreal,  Quebec,  Canada.  This  paper  is  provided  as  a 
supplement  to  this  report. 

Another  effort  was  also  put  forward  to  further  understand  and  model  the  behavior  of  the 
spinning  disk  IPCA  with  respect  to  the  codes  that  it  generates.  The  generation  of  a 
procedure  for  producing  hole  patterns  that  lead  to  independent  codes  are  of  interest  as 
independent  codes  improve  the  image  quality.  Initial  work  in  this  regard  was  published  in 
E,  Jacobs  and  O  Furxhi,  "Image  Plane  Coding  for  Terahertz  Imaging,"  in  Optical 
Sensors,  OSA  Technical  Digest  (online)  (Optical  Society  of  America,  2012),  paper 
SW3C.3,  Monterey,  CA,  USA.  This  paper  is  provided  as  a  supplement  to  this  report. 

In  summary,  the  research  effort  has  advanced  the  terahertz/sub-millimeter  wave  imaging 
device  from  a  line  imager  with  offline  image  reconstructions  to  a  two-dimensional,  real¬ 
time  imager  with  50Hz  image  scan  rates  and  real-time  reconstructions.  The  effort  has  also 


generated  further  analysis  on  this  imaging  approach  and  has  identified  aspects  of  the 
approach  that  need  improvement. 
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7.0  Turbulence  Mitigation 

Task  7.1:  Investigate  the  functional  applicability  and  limitations  of  current 
turbulence  mitigation  techniques  and  algorithms. 

In  support  of  Task  7.1  a  thorough  review  of  the  most  common  mitigation  techniques  was 
completed,  their  respective  algorithms  implemented,  and  the  results  compared.  The 
comparison  resulted  in  the  discovery  of  a  number  of  advantages  and  disadvantages 
related  to  the  implementation  of  each.  They  may  be  summarized  as  follows. 
Lucky/synthetic  imaging  is  based  on  the  finite  probability  that  an  uncorrupted  frame  and 
will  eventually  propagate  through  the  turbulent  atmosphere.  It  has  of  the  advantage  of 
automatically  handling  isoplanatic  restriction  that  plagues  most  of  the  other  turbulent 
mitigation  techniques  and  enables  diffraction  limited  imaging.  The  disadvantages  of  the 
algorithm  lie  in  that  it  depends  on  short  exposure  imagery,  calculation  of  image  quality 
metrics,  and  dependence  on  a  mosaic  procedure  for  best  results  even  though  isoplanatism 
is  handled  inherently  in  the  algorithm.  Blind  deconvolution  has  the  advantage  that  it  can 
be  implemented  for  correction  of  a  wide  variety  of  turbulent  atmospheres  even  when  little 


detail  of  the  intervening  atmosphere  is  provided.  The  disadvantages  include  the  enormous 
computational  complexity,  no  guarantee  of  convergence  to  the  optimal  solution,  and 
dependence  on  isoplanatic  patch  size.  Higher  order  spectral  techniques  have  the 
advantage  of  being  spectrally  intuitive,  resistant  to  corruption  from  white  Gaussian  noise, 
capable  of  identifying  nonlinear  relationships  between  phase  components  and  therefore 
enabling  a  better  representation  of  the  degraded  object.  The  drawbacks  to  its 
implementation  are  the  significant  data  storage  requirement,  trial  and  error  determination 
of  the  intervening  atmosphere’s  spectral  amplitude,  the  dependence  on  isoplanatic  patch 
size,  and  the  inherent  fundamental  frequency  linear  phase  corruption.  Finally,  frame 
averaging  was  the  final  mitigation  technique  investigated.  It  has  the  advantage  of  being 
simple,  computationally  efficient,  and  effective  in  the  removal  of  the  jitter  associated 
with  a  turbulent  atmosphere.  The  drawbacks  to  its  implementation  are  that  it  effectively 
prevents  diffraction  limited  imaging  and  is  not  suitable  for  targets  traveling  with 
considerable  speed.  These  results  appear  as  an  unpublished  paper  and  further  details 
related  to  the  implementation  can  be  found  in  Appendix  X. 


Task  7.2:  Examine  the  turbulent  effects  as  a  function  of  range,  atmospheric 
effects,  imaging  band,  optical  design,  and  detector  type. 

In  support  of  Task  7.2,  an  efficient  turbulence  simulation  algorithm  was  developed  to 
reproduce  the  turbulent  atmosphere  effects  on  a  pristine  image  or  video  of  an  applicable 
target.  The  algorithm  is  based  on  visible  wavelength  statistics  captured  from  various 
turbulent  atmospheres.  This  effort  extended  those  results  to  include  infrared  wavelengths. 
Additionally,  the  simulation  tool  allows  the  user  to  input  range,  strength  of  turbulence, 
imaging  band,  short  or  long  exposure  detector  integration  times,  and  other  optical  design 
characteristics.  The  result  is  a  simulated  image  or  image  sequence  representing  the  effects 
of  the  atmosphere,  the  optics,  and  the  detector.  The  degraded  imagery  is  suitable  for  most 
types  of  turbulence  mitigation  algorithm  development.  It  should  be  noted  that  the 
simulation  algorithm  has  not  undergone  strenuous  comparison  to  real  collected  data.  That 
effort  is  currently  underway.  The  details  of  the  implementation  and  the  result  results  from 
this  effort  have  been  published  and  can  be  found  in  Appendix  X. 

Task  7.3:  Explore  the  effects  of  turbulence  on  hyper-spectral  data,  fusion 
and  metric  development  and  the  effects  of  mitigation  on  feature 
discrimination. 

The  following  efforts  were  undertaken  to  determine  the  effects  of  turbulence  on  feature 
extraction  techniques  applied  to  a  sparse  detector  sensor  arrays.  Video  data  of  humans 
and  animals  was  captured  using  a  long  wave  infrared  imaging  sensor  array.  That  data  was 
subsequently  processed  to  extract  profile  data,  determine  applicable  height  to  width 
ratios,  and  classified  based  on  the  extracted  features.  The  turbulence  simulation  algorithm 
developed  in  support  of  Task  7.2  was  then  used  to  corrupt  the  same  imagery  The  height 
to  width  ratio  of  the  resultant  imagery  was  examined  to  determine  the  quantitative 
differences  when  compared  to  the  uncorrupted  sequences.  The  results  show  that  for 
coarse-grained  classifications  such  as  human  versus  animal  determination,  even  the 
strongest  levels  of  turbulence  only  minimally  affect  the  height  to  width  ratios.  Thus,  we 


can  conclude  that  for  human  versus  animal  classification,  turbulence  should  not  be  much 
of  an  issue.  However,  as  noted  above,  the  turbulence  simulation  algorithm  is  not 
undergone  strenuous  data  verification  and  thus  the  application  of  these  results  should  be 
limited  until  such  time  that  verification  has  been  completed.  These  results  appear  as  an 
unpublished  paper  and  further  details  related  to  the  implementation  can  be  found  in 
Appendix  X. 

Task  7.4:  Consider  the  effects  of  the  extension  of  current  turbulence 
mitigation  techniques  to  sparse  detector  sensor  arrays. 

The  efforts  in  support  of  Task  7.4  include  the  consideration  of  current  turbulence 
mitigation  techniques  and  their  application  to  the  profile  extraction  system. 
Implementation  of  a  turbulence  mitigation  algorithm  cannot  be  considered  apart  from  the 
processing  platfonn.  Therefore,  the  recommendations  derived  under  this  section  reflect 
the  fact  that  a  low  power  sparse  detector  array  makes  up  our  sensing  platfonn.  Given 
these  constraints  lucky/synthetic  imaging  and  single  frame  or  multi-frame  de-convolution 
cannot  be  considered  due  to  their  dependence  on  short  exposure  imaging  and  high 
computational  complexity.  Higher  order  spectral  techniques  must  also  be  excluded  due  to 
the  large  amount  of  data  storage  require  for  their  implementation  and  their  heavy 
computational  complexity  as  well.  Therefore  we  are  left  with  only  frame  averaging  as 
applicable  mitigation  technique  for  the  sensing  platform.  Frame  averaging  represents  a 
simple  and  straightforward  method  of  improving  the  captured  imagery  for  human 
consumption  and  can  be  implemented  quite  easily  on  most  low  power  platforms. 


Papers  Published  in  Support  of  Project  (Copies  provided  in 
Appendix  X) 


[Robinson  2012]  “An  efficient  turbulence  simulation  algorithm”,  A.  L.  Robinson,  J. 
Smith,  and  A.  Sanders,  Proc.  SPIE  8355,  83550K  (2012),  DOI:10. 1 1 17/12  919541 


8.0  Feature  Sensing  Laboratory 

Task  8.1:  Characterize  the  existing  campus  sensor  system  including  a 
formal  knowledge  model  captured  in  an  ontology.  This  will  involve  learning 
what  is  currently  present  on  campus  with  regard  to  sensing  systems, 
instrumenting  them  so  that  they  produce  data  instead  of  just  pictures,  and 
establish  a  means  for  archiving  this  data. 

Task  8.2:  Supplementing  the  existing  campus  sensor  system.  These 
sensors  will  either  be  sensors  operating  in  non-visible  wavebands  or 
sensors  designed  to  provide  new  sensing  capabilities  such  as  profiling 
sensors. 

Task  8.3:  Experiment  design.  We  will  design  multi-use  experiments  in 
which  scenarios  are  created  to  stimulate  events  of  interest. 

An  important  question  in  the  development  of  unmanned  ground  sensors  (UGS)  is  what 
the  field  performance  will  be  long  term  and  under  different  environmental  conditions.  It 
would  be  desirable  to  have  a  long-term  emplacement  of  sensors  that  could  be  monitored 
24/7  in  order  to  gain  data  on  real  world  performance  of  these  systems.  As  part  of  an 
Army  sponsored  program  on  Intelligent  Network- 
Centric  Sensors,  we  have  begun  building  an  UGS  field 
test  laboratory.  When  fully  built,  studies  as  outlined  in 
the  tasks  above  could  be  carried  out. 

Although  initially  planned  as  an  on-campus  facility,  it 
was  decided,  in  consultation  with  the  government 
technical  POC,  to  implement  this  laboratory  in  an 
environment  more  typical  for  border  and  perimeter 
security  applications.  The  site  we  have  chosen  is  a 
biological  field  station  owned  by  the  University.  The 
site  chosen  is  the  University  of  Memphis  Meeman 
Biological  Field  Station.  The  site  covers  623  acres  and 
is  located  approximately  twenty  five  miles  northwest 
fonn  the  main  university  campus.  The  University  of 
Memphis  operates  an  earthquake  monitoring  station 
which  includes  a  microwave  link  back  to  the  campus 
(see  Figure  8.1). 

Our  design  allows  the  status  and  information  collected 
by  sensors  to  be  reviewed  at  the  monitoring  station  in 
near  real  time.  The  data  collected  is  moved  through 
routing  hardware  and  piped  through  a  local  network  to 
a  the  microwave  link.  This  RF  link  consists  of  a  300  ft  Figure  8.1.  RF  antenna  -  Meeman  forest 
antenna  and  a  robust  4.2  GHz  Freewave  link  direct  to 

the  campus  network.  The  antenna  and  RF  equipment  are  located  approximately  900  ft 
from  the  test  site.  Once  the  information  arrives  at  the  main  campus  it  is  routed  to  the 
appropriate  destination  via  secure  links. 


The  laboratory  is  configured  to  allow  deployment  of  static  and  mesh  sensor  networks. 
This  configuration  is  centered  on  a  base  station 
that  is  in  charge  of  routing  stand  alone  and 
mesh-network  data  to  the  RF  link.  The  base 
stations  can  also  act  as  sensor  test  point.  The 
implemented  architecture  also  has  the  flexibility 
to  accommodate  multiple  base  stations  if 
necessary.  This  means  that  any  desired 
combination  of  sensor  network  can  be  easily 
implemented  and  deployed.  A  diagram  of  the 
network  layout  is  shown  in  Figure  8.2. 


Mesh  networks  based  on  XBee  ZigBee  modules 
and  static  nodes  using  XBee  Wi-Fi  modules 
were  tested.  These  modules  were  chosen 
because  they  offer  a  simple  standard  interface 
and  their  low  power  usage  (see  Figure  8.3).  The 
drawback  is  the  speed  limitation  imposed  by  the 
serial  interface.  A  maximum  speed  of 
240Kbaud  limits  the  amount  of  usable 
information  that  can  be  transmitted.  On  average, 
a  1.2MB  data  file  will  take  1  minute  to  transfer. 

This  makes  this  configuration  ill  suited  for  video  but  highly  efficient  to  monitor  low  data 
rate  sensors,  such  as  profiling  sensors.  The  network  can  be 
configured  to  use  standard  Wi-Fi  in  order  to  achieve  higher 
speeds.  This  change  however,  requires  that  additional  steps  be 
taken  to  maximize  power  efficiency  of  deployed  sensor  modules 
in  order  to  extend  their  up- time. 

Several  types  of  local  hardware  configurations  for  sensor  control 
were  evaluated.  They  range  from  the  simplest  “sensor  atom” 
composed  of  an  Arduino  microcontroller,  a  pyroelectric  detector 
and  an  XBee  radio  to  a  base  station.  The  basic  base  station  is 
single  board  computer.  This  computer  is  typically  based  on  and  Atom  processor  and  runs 
Linux.  This  provides  the  flexibility  and  power  of  a  full  fledged  PC  computer  but  drawing 
a  maximum  power  of  40Watts.  If  higher  power  efficiency  is  required  this  can  be  scaled 
down  to  an  Arm  processor  computer  such  as  a  beagle  board. 

Preliminary  test  indicate  that  with  line  of  sight  or  near  line  of  site  placement  of  sensor 
nodes  communications  at  over  1500ft  can  be  easily  maintained.  This  suggests  that  for 
practical  deployment  inter  sensor  node  spacing  of  300  to  500  feet  is  sustainable.  The 
clear  implication  is  that  a  substantial  amount  of  terrain  can  be  monitored  with  relatively 
few  sensor  nodes.  On  the  other  hand  base  stations  deployed  as  direct  gateways  to  the  RF 
link  must  be  deployed  within  line  of  site  of  it  to  maintain  reliable  communications. 


Figure  8.3.  Xbee 
module 
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Figure  8.2.  Network  layout 


Our  initial  deployment  of  the  systems  is  shown  in  this  diagram.  We  have  chosen  to  use  an 
open  source  Gentoo  based  platform  for  the  sensor  base  station.  This  will  provide  control 
of  most  of  the  Meeman  based  communications.  To  this  will  be  connected  high  bandwidth 
sensors  such  as  forward-looking  infrared  camera  and  a  low  bandwidth  sensor  network. 
For  this  project,  two  FLIR  Photon  640  sensors  were  obtained  for  configuration  in  the 
sensor  network.  The  low  bandwidth  network  will  be  a  ZigBee  sensor  network  consisting 
of  MICAz  motes  with  attached  pyroelectric  motion  sensors  deployed  along  a  trail  to 
monitor  movement  of  animals  and  people.  The  MICAz  motes  allow  for  basic  processing 
and  present  the  possibility  of  just  transferring  the  results  of  classifications.  A  distributed 
array  of  motion  detecting  sensors  has  been  shown  in  some  of  our  research  to  be  highly 
effective  in  classifying  humans  and  animals.  We  will  deploy  a  PIR  based  version  of  this 
array  along  the  trail.  The  initial  array  will  consist  of  only  three  sensors  but  can  be 
expanded.  Studies  have  shown  that  very  limit  classification  can  be  made  with  a  single 
sensor.  Our  hope  is  to  develop  a  three-sensor  system  with  better  performance. 

Many  of  the  stated  tasks  for  this  portion  of  the  research  were  not  achieved  due  to  delays 
and  complications  in  implementing  the  network  involved.  Significant  time  was  spent 
tracking  down  communication  interference  between  our  network  router  and  the 
earthquake  sensors.  This  interference  was  resulting  in  false  reports  of  earthquake  activity. 
A  significant  effort  at  providing  additional  shielding  for  our  router  seems  to  have  solved 
the  problem  and  we  have  made  significant  progress  this  summer. 

We  expect  to  have  our  initial  version  of  the  network  up  and  running  by  the  first  of 
September  2012.  Over  time  we  expect  to  expand  the  number,  variety,  and  coverage  of 
sensors  in  the  network.  This  ultimately  may  require  a  dedicated  landline  or  additional 
microwave  link  to  support  the  bandwidth  necessary.  We  plan  to  use  this  facility  to 
perform  testing  of  new  sensors  and  algorithms,  energy  harvesting  research,  and  smart 
tasking  of  sensors  via  intelligent  networks. 


9.0  Conclusions 


The  execution  of  this  contract  has  resulted  in  some  significant  scientific  and  technical 
progress  in  support  of  intelligent,  network-centric,  sensors.  We  believe  that  this  work  has 
been  of  immediate  benefit  to  the  Army  and  will  continue  to  pay  dividends  in  technical 
achievement  into  the  near  future.  Further  publications  will  result  directly  and  indirectly 
from  this  work. 


Appendix  A.  Published  papers  in  support  of  this 
contract. 


Copies  of  all  published  papers  completed  as  part  of  this  cooperative  agreement  are 
provided  following  this  page. 
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ABSTRACT 

The  notion  of  a  profiling  sensor  was  first  realized  by  a  near-IR,  retro-reflective  prototype  consisting  of  a  vertical 
column  of  sparse  detectors.  Alternative  arrangements  of  detectors  have  been  implemented  in  which  a  subset  of  the 
detectors  have  been  offset  from  the  vertical  column  and  placed  at  arbitrary  locations  along  the  anticipated  path 
of  the  objects  of  interest.  All  prior  work  with  the  near-IR,  retro-reflective  profiling  sensors  has  consisted  of  wired 
detectors.  This  paper  advances  this  prior  work  by  designing  and  implementing  a  wireless  prototype  version  of  a 
near-IR,  retro-reflective  profiling  sensor  in  which  each  detector  is  a  wireless  sensor  node.  In  this  novel  architecture, 
a  base  station  is  responsible  for  collecting  all  data  from  the  detector  sensor  nodes  and  coordinating  all  pre¬ 
processing  of  data  collected  from  the  sensor  nodes,  including  data  re-alignment,  before  subsequent  classification 
algorithms  are  executed.  Such  a  wireless  detector  configuration  advances  deployment  options  for  near-IR,  retro- 
reflective  profiling  sensors. 

Keywords:  Wireless  profiling  sensor,  wireless  sensor,  neural  network,  object  classification. 

1.  INTRODUCTION 

The  original  notion  of  a  profiling  sensor  was  proposed  by  Ronnie  Sartain  of  the  U.S.  Army  Research  Laboratory, 
in  which  objects  traversing  the  field  of  view  of  a  crude  imaging  sensor,  consisting  of  a  collection  of  sparse  detectors, 
could  be  classified  by  examining  their  silhouettes.1  A  vertical  array,  consisting  of  sixteen  pairs  of  transmitters 
and  reflectors,  was  first  developed  to  produce  sixteen  parallel  near-IR  beams  (Figure  1).  A  silhouette  was  then 
generated  if  the  object  traversing  the  specified  trail  would  break  the  beams1-5  (Figure  2).  It  was  proposed  that 
a  low-cost  sensing  device  of  this  form  would  be  useful  for  providing  security  surveillance  for  several  applications, 
including  the  U.S. -Mexican  border,  to  detect  illegal  activity  and  in  other  areas  where  the  classification  of  objects 
is  highly  relevant.6  Because  of  the  extent  of  the  U.S.-Mexican  border,  using  low-cost  surveillance  equipment 
would  make  it  economically  feasible  to  cover  more  areas  than  using  traditional  imaging  sensing  devices. 

The  first  profiling  sensor  involved  a  vertical  array  of  sixteen  near-IR  retro- reflective  sensors  that  generated  a 
crude  image.7  Each  sensing  element  produced  a  near-IR  retro-reflective  beam  perpendicular  to  a  pairing  reflecting 
surface  mounted  on  an  opposing  pole.3,8,9  A  classification  algorithm  would  then  take  the  raw  data  (broken  or 
unbroken  beam)  generated  by  the  profiling  sensor  to  produce  and  classify  an  object’s  silhouette  into  one  of  three 
categories  of  interest  (i.e.  human,  animal,  or  vehicle).8,9  As  a  next  step  to  improve  this  prototype’s  versatility, 
a  low-resource  microcontroller  was  interfaced  with  this  profiling  sensor.  The  Digi  International  BL4S200  single¬ 
board-computer  was  used  to  store  and  classify  data  of  the  profiling  sensor.10  Other  hardware  improvements,  such 
as  an  interface  handheld  I/O  box,  were  added  later  on  to  facilitate  data  access  to  the  microcontroller  without 
the  need  of  a  computer.9  Furthermore,  an  additional  improvement  included  the  arrangement  of  sensing  elements 
into  a  custom  arrangement9  (Figure  3)  while  still  generating  similar  raw  data  that  could  be  discriminated  among 
the  three  classifications  groups  of  interest.11  This  new  offset  arrangement  of  detectors  may  provide  a  means  by 
which  the  sensor  could  be  more  easily  concealed. 
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Figure  1.  Object  passing  through  the  parallel  beams  of  a  vertical  array  of  a  near-IR  retro- reflective  profiling  sensor. 


Figure  2.  Generated  silhouette  of  a  human  after  traversing  through  the  profiling  sensor. 

This  paper  focuses  on  a  wireless  near-IR  retro- reflective  profiling  sensor  that  can  be  thought  of  as  a  set  of 
sixteen  wireless  sensing  nodes  that  comprise  the  overall  profiling  sensor.  Previous  profiling  sensor  prototypes  have 
used  CX-RVM512  active  near-IR  detectors  that  depend  on  their  own  paring  reflector.  Each  node  in  the  wireless 
profiling  sensor  uses  a  Sharp  GP2Y0D02YK0F13  (Figure  4)  as  the  sensing  element  with  a  similar  field  of  view 
to  the  CX-RVM5  but  does  not  require  a  reflector  to  sense  a  change-of-state  event.  The  Sharp  GP2Y0D02YK0F 
sensing  element  can  thus  be  thought  as  a  touch-less  switch  that  turns  on  (digital  high)  if  an  object  obstructs 
the  path  of  its  infrared  beam  and  turns  off  (digital  low)  if  there  is  no  obstruction.  Additionally,  to  make  the 
sensing  node  wireless,  each  sensing  element  has  been  hardwired  to  its  own  wireless  transmitter:  an  OEM  Radio 
Frequency  (RF)  board14  (Figure  5).  A  sensing  node  sends  one  packet  containing  the  state  and  node  identification 
data  to  a  USB  gateway  receiver  only  if  a  change  of  state  occurs  (high-to-low  or  low- to- high)  as  an  object  passes 
through  its  field  of  view.  The  computer  recording  the  data  can  then  assign  a  binary  value  to  each  state  and 
generate  a  silhouette  representation  of  the  object  or  process  it  by  the  classification  algorithm  described  in  the 
subsequent  sections. 

The  focus  of  this  paper  is  to  present  the  architecture  and  the  interaction  of  a  wireless  profiling  sensor  with 
a  computer.  The  computer  software  developed  for  this  work  integrates  useful  tools,  such  as  object  classification 
and  object  silhouette  previews,  which  can  be  provided  in  a  test  scenario  immediately  after  each  passing  object  is 
detected.  A  trained  classifier  implemented  by  a  bade- propagation  neural  network  is  also  introduced  and  shown 
to  provide  classification  results  of  94%  accuracy  in  testing  to  date. 
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Figure  3.  Object  passing  through  the  beams  of  a  custom  array  near-IR  retro-reflective  profiling  sensor. 


Figure  4.  Sharp  GP2Y0D02YK0F  distance  measuring  sensor  unit. 


Figure  5.  OEM  RF  board  (Left)  and  USB  gateway  receiver  (Right). 


2.  DATA  LIBRARY 

In  previous  work  done  by  Russomanno  et  al.  at  the  University  of  Memphis,  a  near-IR  profiling  sensor  was  used 
to  capture  profiles  of  over  1000  objects  in  the  field,  including  humans,  animals,  and  vehicles.15  Each  sample 
was  captured  at  approximately  1  kHz  and  stored  as  a  matrix  of  16  binary  strings  corresponding  to  each  of  the 
16  sensing  elements  in  the  array.10  The  length  of  each  binary  string  depended  upon  the  velocity  at  which  the 


object  traveled  through  the  specified  field  of  view.9’16*17  In  other  words,  a  vehicle  traveling  at  a  higher  velocity 
would  correlate  to  shorter  binary  strings,  while  a  vehicle  traveling  at  a  lower  velocity  would  3d  eld  longer  strings. 

Since  the  profiling  sensor  generates  outputs  comprised  of  zeroes  and  ones,  it  is  especially  convenient  to  use 
a  back-propagation  neural  network  as  it  preferably  processes  data  that  has  been  normalized.  However,  the 
architecture  of  the  back-propagation  neural  network  requires  a  fixed  number  of  input  arguments;  therefore,  it  is 
necessary  to  compress  all  data  into  a  fixed  column  dimension  size.  Thus,  for  this  approach,  an  algorithm  has 
*  been  used  to  achieve  this  defined  length  without  significantly  altering  the  data  in  the  sample  library.  Figure  6 
shows  silhouette  images  before  and  after  the  compression  algorithm  has  been  applied.  After  analyzing  the  same 
compression  results  in  the  remainder  of  the  library  samples,  it  was  determined  that  no  major  resolution  loss 
occurs  if  these  are  compressed  to  256  columns.  Therefore,  a  new  database  of  silhouettes  was  generated  with  the 
compressed  samples  to  use  as  the  training  and  testing  data  set  of  the  back-propagation  neural  network. 
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Figure  6.  Silhouettes  before  compression  and  after  compression:  (a)  Human  silhouette  before  compression,  (b)  Human 
silhouette  after  compression,  (c)  Vehicle  silhouette  before  compression,  (d)  Vehicle  silhouette  after  compression,  (e)  Animal 
silhouette  before  compression,  (f)  Animal  silhouette  after  compression 


In  the  original  near-IR  retro-reflective  profiling  sensor,  each  sensor  recorded  one  bit  per  sampling  cycle.16, 17 
This  generated  a  real-time  recording  of  the  obstruction  and  restoration  of  each  sensor  beam.  In  the  wireless  near- 
IR  retro-reflective  profiling  sensor,  individual  transmitters  send  one  packet  for  each  change  of  state  in  a  sensing 
node  (object  or  no  object  obstruction)  instead  of  frequently  sampling  the  current  state  of  each  sensing  element. 
This  method  reduces  power  consumption14  compared  to  the  current  model,  which  extends  the  useful  life  of  the 
device.  A  wireless  sensor  node  only  sends  a  packet  at  each  change  of  state:  one  packet  for  obstruction  and  one 


packet  for  restoration  of  the  beam.  Using  the  time  difference  between  changes  of  states,  a  computer  algorithm 
generates  the  zeros  and  ones  correlating  to  the  obstruction  and  restoration,  forming  the  16  binary  strings  by  a 
previously  defined  length  (256  bits).  Though  the  recording  method  is  different  than  the  original  sensor  in  this 
new  prototype,  each  of  the  sensing  nodes  has  a  comparable  field  of  view  as  the  original  wired  prototype,  which 
does  not  affect  the  resolution  of  the  recording  and  results  in  similar  silhouette  data  as  presented  in  Figure  7. 


Figure  7.  Human  silhouette  generated  by  the  wireless  near-IR  retro-reflective  profiling  sensor. 


3.  WIRELESS  SENSING  NODE  HARDWARE 

In  an  effort  to  make  the  new  profiling  sensor  independent  from  a  pairing  reflective  surface,  a  near-IR  distance 
sensing  device  was  used  to  detect  objects  along  the  path  of  each  sensing  element.  This  sensing  element  is  a  single 
point  distance  module,  which  measures  the  distance  from  the  emitting  infrared  diode  to  the  reflecting  surface  of 
the  object.  Furthermore,  the  design  of  the  sensing  element  used  in  this  prototype  is  not  considerably  affected 
by  environmental  light  or  the  reflective  properties  of  the  object  up  to  a  guaranteed  distance  of  80cm.13  The 
near-IR  distance  sensor  module  utilizes  the  triangulation  principle:  the  laser  path  is  initiated  at  the  near-IR 
beam  emitter,  reflects  off  of  the  object,  and  is  captured  by  the  imager.  When  this  event  occurs,  the  output  line 
of  the  sensor  is  set  to  a  logical  high,  otherwise  is  set  to  low.  The  relationship  between  the  emitter,  object,  and 
imager  is  shown  in  Figure  8.  The  horizontal  distance  from  the  emitter  to  the  object  relies  on  the  following:  1) 
the  angle  created  between  the  emitted  laser  path  to  the  object  and  the  reflected  laser  path  to  the  imager;  and  2) 
the  distance  between  the  IR  beam  and  the  imager.  The  horizontal  distance  can  be  easily  calculated  by  Equation 
1,  while  Equation  2  utilizes  the  sine  of  the  angle  alpha  to  determine  the  horizontal  distance.18 
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Figure  8.  Relative  distances  and  angle  between  object  and  individual  elements  in  the  distance  sensor. 
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To  support  wireless  data  transmission,  each  of  the  16  distance  sensing  elements  has  been  interfaced  to  their 
own  OEM  wireless  transmitter.  The  OEM  wireless  transmitter  has  been  programmed  to  send  a  packet  of  data 
when  a  change  of  logic  state  is  received  by  the  distance  sensing  element.  A  base  station  then  gathers  the  data 
packets  from  the  16  wireless  near-IR  retro- reflective  sensing  nodes  and  processes  them  for  object  classification 
and  silhouette  image  generation.  Figure  9  shows  the  wireless  near-IR  retro-reflective  profiling  sensor  system 
made  up  of  16  near-IR  wireless  sensing  nodes  and  one  base  station  that  receives  the  packets  from  each  node. 
However,  specific  I/O  requirements  between  the  OEM  wireless  transmitters  and  the  near-IR  distance  sensing 
device  required  additional  hardware  that  would  not  only  need  to  regulate  the  voltage  to  each  device,  but  the 
signal  going  from  the  sensing  element  to  the  transmitter.  Figure  10  shows  a  block  diagram  representation  of  one 
near-IR  wireless  sensing  node. 
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Figure  9.  Wireless  near-IR  retro-reflective  profiling  sensor  made  up  of  16  near-IR  sensor  nodes.  A  base  station  collects 
the  packets  from  each  change  of  state  event  as  an  object  crosses  through  the  IR  beams. 


The  OEM  wireless  transmitters  have  been  used  in  this  wireless  profiling  sensor  because  they  can  be  easily 
interfaced  with  external  digital  and  analog  devices.  Finally,  each  transmitter  supports  multiple  transmission 
frequencies  options:14  the  ISM  Band  902-928MHz  has  been  used  in  the  complete  wireless  near-IR  retro- reflective 
profiling  sensor. 


4.  DATA  ACQUISITION 

To  gain  adequate  data  to  identify  a  passing  object,  the  status  of  each  node  must  only  be  known  at  two  stages: 
when  the  object  first  obstructs  the  beam  and  when  the  beam  is  restored.  Previous  profiling  sensor  prototypes 
utilize  a  constant  collection  of  data  at  a  set  frequency  for  its  method  of  data  acquisition.16, 17  As  previously 
mentioned  in  section  2,  a  similar  data  acquisition  method  would  quickly  drain  the  battery  of  the  wireless  pro¬ 
filing  sensor  since  the  transmitter  needs  to  pull  current  for  every  packet  transmission.14  Therefore,  we  have 
implemented  a  data  acquisition  method  that  not  only  uses  the  battery  more  efficiently,  but  only  uses  one  data 
packet  per  change  of  state  from  each  sensing  element  to  generate  the  necessary  raw  data. 
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Figure  10.  Wireless  near-IR  retro-reflective  sensor  node  block  diagram. 


In  the  wireless  profiling  sensor,  there  is  a  designated  trigger  node  that  alerts  the  base  station  to  the  presence 
of  an  object.  When  this  node  detects  an  obstruction,  the  base  station  will  begin  recording  data  from  all  sensing 
nodes  for  a  period  of  time  specified  in  sensor’s  configuration  setup.  This  packet  contains  the  state  of  the  node 
and  the  source  node  identification  number.  The  state  of  the  node  is  a  binary  code  in  which  zero  (0)  indicates 
an  unbroken  beam  and  one  (1)  indicates  an  obstructed  beam.  As  the  base  station  gathers  data  packets,  it  will 
also  record  the  time  in  milliseconds  of  when  each  packet  is  received.  After  the  specified  recording  time  expires, 
the  base  station  ceases  to  record  data.  The  recording  of  one  object  during  this  period  of  time  is  denoted  as 
one  recorded  event.  The  base  station  sorts  the  collected  transmission  records  of  the  recorded  event  by  source 
node  identification  numbers.  Then  an  algorithm  determines  the  time  length  of  the  recorded  event  by  calculating 
the  difference  between  the  time  stamp  of  the  final  packet  received  and  the  time  stamp  of  the  initial  packet 
received.  In  previous  prototypes,  time  stamps  were  unnecessary  because  of  the  continuous  data  recording,  but 
since  this  new  prototype  collects  data  only  when  a  change  of  state  occurs,  we  must  generate  the  frequency  of 
data  acquisition  for  our  non-constant  rate  to  develop  a  binary  matrix  representation  of  this  data. 

The  resulting  binary  matrix  is  a  16  x  m  matrix  in  which  every  entry  corresponds  to  a  low  or  a  high  bit  reading 
at  a  point  in  the  sampling.  To  generate  this  matrix  with  the  collection  of  time  samples  received,  the  total  time 
length  of  the  recording  A t  is  calculated  by  subtracting  the  time  of  the  first  packet  received  4  from  the  time  the 
last  packet  is  received,  if.  Equation  3  is  then  used  to  find  the  length  of  a  time  segment  that  will  eventually  be 
replaced  by  a  binary  value  corresponding  to  the  node  state  in  that  time  interval.  Additionally,  4  —  ta  is  the  time 
difference  from  a  change  of  state  within  At.  As  shown  in  Equation  4,  4  —ta  is  divided  by  the  result  in  Equation 
3  so  that  this  time  interval  can  be  interpreted  as  a  sequence  of  bits  describing  the  constant  state  of  the  (4 >4) 
interval.  Figure  11  is  a  visual  representation  of  this  procedure.  Repeating  this  procedure  for  each  sensing  node 
generates  the  16  necessary  binary  rows  making  up  the  entire  binary  matrix  of  the  silhouette.  Finally,  this  matrix 
is  then  used  to  generate  a  silhouette  image  of  the  passing  object,  as  well  as  input  data  to  a  back-propagation 
neural  network  for  its  classification. 
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Figure  11.  Populating  a  binary  row  by  using  a  change- of- state- time-segmentation  method. 


5.  CLASSIFICATION  AND  RESULTS 

Although  the  original  raw  data  had  varying  column  dimensions,  it  has  been  shown  that  compressed  data  with 
a  fixed  number  of  columns  produces  virtually  no  alterations  in  the  silhouette  resolution  in  the  analyzed  library. 
Since  a  bade- propagation  neural  network  is  used  to  classify  the  data,  a  fixed  number  of  inputs  were  required  to 
obtain  a  classification  result.  After  normalizing  the  data  to  256  columns,  the  back-propagation  neural  network  was 
trained  with  half  the  data  set  within  the  compressed  library.  This  neural  network  is  comprised  of  256  x  16  =  4096 
processing  elements  in  the  input  layer,  20  processing  elements  in  the  hidden  layer,  and  3  processing  elements  in 
the  output  layer.  This  configuration  was  observed  to  provide  the  best  classification  results  compared  with  other 
back-propagation  neural  networks  with  differing  numbers  of  processing  elements  within  the  hidden  layer.  The 
method  used  to  train  the  neural  network  was  from  Ebehart  and  Shi.19 


Figure  12.  Matrix  vectorization:  all  rows  from  a  two-dimensional  matrix  are  put  sequentially  into  a  linear  array. 


To  make  the  matrix  easier  to  handle  by  the  neural  network,  the  binary  matrix  has  been  vectorized,  a  process 
in  which  all  rows  are  combined  sequentially  (Figure  12).  Each  processing  element  in  the  input  layer  uses  each 
bit  entry  in  the  vector  matrix  as  the  argument  to  a  sigmoid  function  (Equation  5),  which  will  generate  a  real 
value  between  0  and  1.  The  output  values  from  each  processing  element  in  the  previous  layer  become  the  input 
values  for  each  of  the  processing  elements  in  the  subsequent  layer.  The  edges  connecting  the  processing  elements 
from  one  layer  to  its  subsequent  layer  have  a  weight  value  which  is  generated  during  the  training  process  of  the 
neural  network.  The  weight  of  the  edge  is  then  multiplied  by  the  input  for  each  processing  element  in  the  hidden 
layer.  The  sum  of  these  4096  products  becomes  the  argument  for  the  sigmoid  function  of  the  hidden  layer.  This 
process  is  repeated  from  the  hidden  layer  to  the  output  layer  to  generate  3  output  values.  A  schematic  of  this 
neural  network  is  shown  in  Figure  13.  Equations  6  and  7  are  used  to  calculate  the  input  values  to  the  processing 
elements  in  the  hidden  and  output  layer.  The  values  x  and  Wji  denote  the  weight  values  from  the  input  to  the 
hidden  layer  and  from  the  hidden  to  the  output  layer,  respectively,  all  which  are  generated  during  the  training 
of  the  neural  network.  Furthermore,  notice  that  the  sum  of  products  begin  from  h  =  0  and  i  =  0,  the  bias 
processing  elements,  which  take  an  input  value  of  1. 
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Figure  13.  Back-propagation  neural  network  with  4096  input  processing  elements,  20  hidden  processing  elements,  and  3 
output  processing  elements.  Bias  processing  elements  are  denoted  by  b  and  these  take  an  input  value  of  1. 
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The -output  values  in  the  output  layer  are  positive  real  numbers  between  0  and  1;  the  output  processing 
element  with  the  highest  value  indicates  the  classification  of  the  input.  A  total  of  577  test  samples  from  three 
classes  were  classified  by  the  back-propagation  neural  network.  The  distribution  of  the  test  samples  from  each 
class  is  shown  in  Figure  14.  When  the  neural  network  assesses  the  entire  set  of  testing  data,  94%  of  the  data 
is  properly  sorted  into  the  correct  classification  groups  (Figure  15).  All  of  this  data  was  previously  collected  by 
the  vertical  sparse  array  profiling  sensor. 
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Figure  14.  Distribution  of  number  of  samples  per  class  within  the  testing  data  set. 

Unlike  the  vertical  sparse  array  profiling  sensor,  the  new  wireless  prototype  allows  for  a  small  horizontal  offset 
of  approximately  5-10  inches.  This  offset  is  achieved  without  significantly  diminishing  the  classification  accuracy. 


However,  to  maintain  this  high  level  of  accuracy,  passing  objects  must  have  an  approximately  constant  velocity.9 
When  a  passing  object  changes  its  velocity  in  the  horizontal  space  between  each  sensing  node,  the  silhouette  will 
be  significantly  distorted  and  classification  results  can  be  affected.  Therefore,  increasing  the  horizontal  spacing 
between  these  nodes  increases  the  risk  of  incorrectly  classifying  the  passing  object.  It  was  determined  that  keeping 
horizontal  distances  within  5  inches  will  correctly  classify  the  data  close  to  94%  accuracy.  For  convenience,  the 
existing  data  library,  comprised  of  data  collected  using  a  vertical  sparse  array  profiling  sensor,  was  used  to  train 
the  back-propagation  neural  network  in  this  experiment.  However,  for  any  configuration  of  sensing  nodes,  a  new 
data  library  can  be  populated  and  used  to  retrain  the  neural  network  to  evaluate  its  classification  performance. 
Finally,  the  confusion  matrix  that  shows  the  performance  of  the  classification  algorithm  in  this  test  is  shown  in 
Table  1. 


Percentage  of  Correctly  and  Incorrectly  Classified  Data  by  Object  Type 
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Figure  15.  Classification  results  (a)  Overall  combined  percent  of  correct  and  incorrect  classification  results,  (b)  Correct 
and  incorrect  classification  results  by  object  classes. 


Table  1.  Confusion  matrix  of  classification  results. 


Actual  Class 


Predicted  Class 


Human 

Animal 

Vehicle 

Human 

306 

20 

7 

Animal 

4 

77 

0 

Vehicle 

0 

0 

163 

6.  CONCLUSION 


By  combining  the  Sharp  GP2Y0D02YK0F  with  an  OEM  RF  board,  a  wireless  profiling  sensor  prototype  was 
designed  and  tested  with  highly  accurate  results  in  classifying  passing  objects  at  close  range.  As  shown  in  these 
experiments,  the  Sharp  GP2Y0D02YK0F  proximity  sensor  can  be  used  to  collect  data  in  a  similar  way  to  the 
CX-RVM5  IR  based  profiling  sensor.  The  data  collected  by  the  proximity  sensors  is  generated  by  using  only 
two  packets  of  data,  one  for  a  broken  beam  and  one  when  the  beam  is  restored.  There  is  no  significant  loss  of 
silhouette  resolution  when  the  normalized  binary  matrix  is  generated.  Furthermore,  this  input  into  the  back- 
propagation  neural  network  is  used  to  train  and  classify  data,  producing  correct  classifications  with  up  to  94% 
accuracy.  In  addition  to  the  increasing  processing  capabilities,  the  flexibility  to  deploy  these  sensors  in  the  field 
is  increased  by  allowing  each  sensing  node  to  have  a  horizontal  offset.  Overall,  this  new  design  of  the  profiling 
sensor  is  an  incremental  improvement  as  compared  to  the  original,  wired  profiling  sensor  prototype. 

In  continuing  this  work,  we  plan  to  collect  data  using  the  wireless  profiling  sensor  using  several  different 
sensing  node  placement  configurations.  This  information  will  be  used  to  retrain  the  back-propagation  neural 
network  and  test  the  classification  performance  of  different  arrangements  of  sensing  nodes.  Additionally,  we  will 
modify  the  OEM  wireless  transmitter  software  to  allow  each  sensing  node  to  calculate  the  relative  position  of  all 
of  the  other  sensing  nodes;  thus  eliminating  the  need  to  train  the  neural  network  for  each  new  configuration  of 
elements.  This  will  ideally  allow  for  completely  randomized  configurations,  rather  than  positioning  the  nodes  in 
set  and  tested  patterns. 
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A bstract- Data -to- decision  systems  must  fuse  information  from 
heterogeneous  sources  to  infer  a  high-level  understanding  of  a 
situation.  A  high  degree  of  confidence  in  the  inferred  knowledge  is 
necessary  for  appropriate  actions  to  be  taken  based  upon  the 
assessment  of  a  situation.  This  paper  presents  an  extensible 
Semantic  Web  compatible  framework  that  uses  rich  ontological 
descriptions  for  the  autonomous  and  human-aided  fusion  of 
heterogeneous  sensors  and  algorithms  to  create  evidence-based 
hypotheses  of  a  situation  under  persistent  surveillance.  Raw  data 
acquired  from  profiling  sensors  is  combined  with  the  output  of 
visualization  and  classification  algorithms,  yielding  information 
with  a  higher  degree  of  confidence  than  what  would  be  obtained 
without  the  fusion  process.  The  framework  can  readily 
accommodate  other  data  sources  and  algorithms  into  the  fusion 
process. 

KEYWORDS:  Semantic  Web,  Ontology,  Sensor  Network,  Data 
Fusion,  Situation  Awareness,  Data  to  Decision  Framework, 
Autonomous  Decision  Systems 

I.  INTRODUCTION 

The  fusion  of  data  from  heterogeneous  sources,  such  as 
sensors  and  intelligence  reports,  is  integral  to  the  inference  of 
highly  reliable,  evidence -based  knowledge  of  a  situation.  The 
degree  of  confidence  in  the  inferred  knowledge  improves  with 
further  acquired  evidence,  that  is,  fusion  of  data  from  more 
sources.  This  paper  describes  an  extensible  Semantic  Web 
compatible  framework  for  the  autonomous  fusion  of  data  from 
heterogeneous  sensors  and  algorithms,  allowing  a  human 
operator  receiving  the  fused  data  to  assess  a  situation  with  an 
increased  confidence  in  the  context  of  persistent  surveillance. 

Situation  awareness  has  been  defmed  by  Endsley  [1]  as 
follows:  “the  perception  of  elements  in  the  environment  within 
a  volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near  future.” 
Situation  awareness  is  critical  to  decision  making  in  many 
applications,  such  as,  patient  monitoring,  emergency  response, 
military  command  control,  and  border  surveillance. 

Systems  for  situation  awareness  require  the  fusion  of  a 
myriad  of  data  and  knowledge  sources,  including  disparate 
sensor  systems,  algorithms,  and  intelligence  reports.  Semi- 
automated  and  automated  inference  using  fused  data  may  lead 
to  an  enhanced  knowledge  about  the  entities  of  interest  in  a 
situation,  as  well  as  an  increased  confidence  in 
interrelationships,  enabling  situation  awareness.  End  user 


confidence  m  the  inferred  knowledge  is  critical  to  timely  and 
appropriate  actions. 

Integration  of  sensor  data  with  algorithmic  processes  and 
human-controlled  information  systems  poses  a  significant 
challenge  for  network-centric  sensor  frameworks.  Fig.  1  is  a 
summary  of  the  classical  Joint  Director  of  Laboratories  (JDL) 
fusion  levels  [2],  augmented  with  a  knowledge  management 
component  [3].  These  six  fusion  levels  cover  both  automated 
and  “human  in  the  loop”  processing  of  data  and  knowledge. 
The  model  supports  the  concept  of  autonomous  algorithms  and 
human  users  contributing  to  an  evolving  solution  state  in  which 
fused  information  may  enable  the  identification  and  assessment 
of  strategies  and  tactics  for  counterintelligence  [3-5].  The  JDL 
model  is  useful  for  describing  the  conceptual  framework  within 
which  a  particular  fusion  process  occurs  and  it  also  provides  a 
reference  for  describing  the  level  of  fusion  in  an  overall 
process. 
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Fig.  1.  Six  levels  of  the  data  fiision  model  augmented  with  a  data  and 
knowledge  management  system. 

Multi-sensor  data  integration  has  been  limited  primarily  by 
the  lack  of  standards  for  data  exchange  and  for  describing 
sensor  capabilities  and  specifications,  which  would  enable  their 
automated  discovery,  invocation,  and  composition  with  other 
sensors  as  part  of  process  workflows  [6-7],  The  XML-based 


data  formats  and  standards,  such  as  SensorML  (Sensor  Markup 
Language)1 *  and  the  Observations  and  Measurements  (O&M) 
Schema4  have  been  adopted  by  the  Sensor  Web  Enablement 
(SWE)  [8]  initiative  of  the  Open  Geospatial  Consortium 
(OGC)3  to  mitigate  these  deficiencies.  The  SWE  initiative  aims 
to  bring  together  Web  connected  sensors  of  all  types  in  a 
common  framework  for  discovery,  invocation,  and  tasking. 
The  advent  of  the  Semantic  Web  initiative  [9]  and  its 
associated  representational  standards,  such  as  the  Web 
Ontology  Language  (OWL)4  and  the  Resource  Description 
Framework  (RDF)5,  has  led  to  the  development  of  sensor 
specific  ontologies,  such  as  OntoSensor  [10]  and  the  Sensor 
Data  Ontology  [11].  These  ontologies  contain  computable 
descriptions  (definitions)  of  sensors  and  sensor  data,  their 
properties,  capabilities,  and  relationships. 

In  this  paper,  we  present  an  onto  logy -based  framework 
that  is  capable  of  fusing  raw  data  with  the  output  of  machine 
learning  algorithms  that  process  the  given  raw  data;  yielding 
further  knowledge  about  a  monitored  situation.  The  framework 
is  robust  and  flexible  to  accommodate  a  variety  of  algorithms 
and  heterogeneous  sensors  by  providing  ontological 
descriptions  of  their  input  and  output  capabilities.  Descriptions 
of  these  components,  which  use  ontologically  defmed  terms, 
allow  capability-based  matching  of  sensors  and  algorithms 
creating  a  plug-and-play  architecture  while  eliminating  the 
hurdles  to  their  interoperability. 

II.  PROFILING  SENSORS  FOR  SURVEILLANCE 

Persistance  surveillance  environments  can  contain  a  wide 
variety  of  sensor  systems  and  algorithms.  One  such  family 
includes  profiling  sensors,  which  are  denoted  by  the 
nomenclature  PFx.  The  PFx  sensors  are  often  regarded  as 
crude  ‘imaging’  devices  and  typically  use  a  sparse  detector 
array,  as  compared  to  a  large  focal  plane  array  found  in 
traditional  cameras  [12].  PFx  sensors  may  be  active  or  passive 
sensors  that  capture  data  about  objects  that  pass  through  their 
field  of  view.  For  example,  Fig.  2-a  shows  an  active  near- 
infrared  version  of  a  profiling  sensor  (PFi)  with  a  sparse  array 
of  detectors  in  a  vertical  geo-location  placement. 


One  algorithm  that  operates  on  PFx  data  has  the  objective 
of  visualizing  the  raw  data  as  a  crude  profile  or  silhouette  for 
human-aided  classification.  In  addition,  other  algorithms 
operate  on  the  PFx  raw  data  and  attempt  to  classify  the  data 
autonomously.  For  example,  one  approach  is  to  attempt  to 
classify  the  data  into  one  of  three  classes:  human,  animal,  or 
vehicle.  Extensions  of  such  algorithms  may  attempt  a  more 
refmed  classification,  such  as,  human  with  backpack,  light 
truck,  etc.  The  PFx  algorithms  may  use  a  variety  of 
classification  techniques,  such  as  the  naive  Bayesian  classifier 
[13],  naive  Bayesian  with  Linear  Discriminant  Analysis  (NB- 
LDA)  for  dimensionality  reduction  [14],  K-Nearest  Neighbor 
(K-NN)  classifier  [15],  Soft  Linear  Vector  Quantization 
(SLVQ)  [16],  and  Support  Vector  Machines  (SVM)  [17]  for 
autonomous  classification  A  comprehensive  set  of  papers  on 
profiling  sensors  appears  in  [18-24]. 

A .  Application  Scenario 

Consider  a  border  patrol  agent  remotely  monitoring  an 
area  for  activity  of  interest.  The  agent  may  be  interested  in 
detecting  humans  attempting  to  cross  a  border  with  or  without 
weapons  and  drugs  using  deployed  PFx  sensor  systems.  In 
many  scenarios,  there  is  a  low  tolerance  for  false  positives, 
given  the  remoteness  of  a  location  under  surveillance  or  the 
possible  hostile  environment.  The  raw  data  output  of  the  sensor 
as  seen  in  Fig.  2b  contains  very  little  information  about  the 
sensed  entity  to  an  end  user.  The  silhouette  generated  from  this 
data,  shown  in  Fig.  2c,  provides  more  information  to  an  end 
user  as  to  the  outline  of  the  detected  entity.  The  agent  is  likely 
to  recognize  the  silhouette  in  Fig.  2-c  as  that  of  a  human. 
However,  some  silhouettes  are  much  less  informative  to  an  end 
user  as  in  Fig.  3.  In  such  cases,  the  silhouette  needs  to  be 
supplemented  with  other  information  to  increase  the  agent’s 
confidence  that  the  object  is  indeed  human  or  a  truck,  animal, 
etc.  A  classification  algorithm  capable  of  processing  the  raw 
data  or  the  silhouettes  may  yield  such  supplemental 
information.  As  a  rudimentary  fusion  scenario,  the  agent  could 
be  provided  with  the  silhouette  in  addition  to  the  output  of  the 
classification  algorithm.  In  many  instances,  the  combined 
information  may  increase  the  agent’s  confidence  in  a  given 
hypothesis. 


Fig.  3.  Examples  of  less  informative  silhouettes  to  end  users 


(a)  (b)  (c) 

Fig.  2.  PF!  sensor  system  comprising  (a)  near  infrared  detectors  in  a  vertical 
geo-looation  placement  (b)  sensor  output  formatted  using  run-length  encoding 
and  (o)  output  of  an  algorithm  that  produced  a  silhouette  of  the  object. 


1http://www.  oppngeospatial.org/standards/sensorml 

7  http://www.oDengeospatial.org/standards/om 

*  httP://www.opengeospatial,org/ 

4  http://www.w3.org/rR/2009/WD-owl2-primer-2009Q611/ 

5  http://www.w3.orE/TR/rdf-primer/ 


III.  FUSION  FRAMEWORK 

The  ontology-based  framework  facilitates  the  integration 
of  raw  data  from  sensor  systems  and  the  output  of  algorithms 
that  process  either  the  raw  data  or  other  intermediary  data 
formats  to  create  high-level  knowledge  of  the  sensed 
environment.  The  cuirent  fusion  framework  desenbed  in  this 
paper  is  an  extension  to  previous  work  of  the  authors  that 
focused  on  autonomous  matching  of  sensor  systems  to 
algorithms  which  in  turn  were  assigned  to  missions  [25-26]. 


A.  Ontology  and  Rule  Framework 

The  framework  was  implemented  using  Description  Logic 
(DL)  ontologies  in  preference  to  conventional  relational 
databases  because  of  the  machine  interpretable  formal  logic 
based  semantics  and  the  inference  capabilities  of  the  former 
The  ontology  (excerpt  as  an  UML  diagram  in  Fig.  4)  was 
created  using  TopBraid  Composer6. 

The  DataFusion  instance  has  been  defined  as  an  aggregate 
of  exactly  one  or  more  DataFusionComponent  instances  which 
may  be  either  an  Image  instance  or  an  AlgorithmOutput 
instance.  The  Image  instance  is  the  product  of  a 
Simp leSensor System  instance,  which  comprises  a  Sensor 
instance  and  an  Algorithm  instance.  The  AlgorithmOutput 
instance  is  the  product  of  an  Algorithm  instance.  Both  the 
Sensor  and  Algoritlvn  classes  are  the  root  of  an  extended 
hierarchy  (not  shown  here,  refer  to  [25]  and  [26])  that  covers 
many  different  sensor  system  types,  such  as  electromagnetic, 
chemical,  and  radar,  as  well  as  algorithmic  processes,  such  as 
profile  generators,  visualizers,  classifiers  and  neural  networks. 


Fig.  4.  Core  ontology  for  sensor  data  fusion  represented  in  UML 


Fig.  5.  An  image  from  a  sensor  system 


Fig.  6.  Fusion  of  the  image  with  the  output  from  another  component,  an  SVM 
classifier  algorithm 


The  philosophy  behind  the  class  diagram  is  explained  in 
Figs.  5-8,  by  the  object  diagrams  adjoining  each  of  the  outputs. 
Fig.  5  shows  a  simple  silhouette  that  has  been  generated  by  a 
sensor  system  comprising  a  PFj  sensor  and  a  visualizing 
algorithm.  A  human  user  may  be  able  to  recognize  this  image 
as  that  of  a  human.  In  comparison.  Fig.  6  shows  the  silhouette 
in  addition  to  the  output  of  a  simple  classifier  algorithm, 
implemented  by  a  Support  Vector  Machine,  which  classifies 
the  image  as  a  human  with  90%  certainty.  Fig.  7  shows  the 
silhouette  supplemented  by  the  output  of  a  Bayesian  classifier, 
which  categorizes  the  silhouette  as  that  of  a  human  with  93% 
certainty.  Lastly,  Fig.  8  illustrates  the  fusion  of  the  outputs  of 
both  classifiers  and  presents  all  the  results  together  with  the 
silhouette.  The  image  and  the  classifier(s)  outputs  are  modeled 
as  components  of  the  Data  Fusion  instance. 
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Fig.  7.  Fusion  of  the  image  with  the  output  from  another  component,  an  NB 
classifier  algorithm 
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Fig.  8.  Fusion  of  the  image  with  the  output  from  multiple  classifiers 


Fig.  10.  SPARQL  query  for  invoking  an  algorithm 


B.  PFx  Example 

We  have  developed  an  application  in  a  persistence 
surveillance  context  to  demonstrate  the  utility  of  the  ontology- 
based  framework  for  the  fusion  of  data  from  sensor  systems 
and  algorithms  The  purpose  of  the  mission  is  to  provide  raw 
surveillance  data  with  algorithmically  derived  profiles  and 
classifications  to  a  human  observer  in  a  remote  location.  Given 
the  ontological  definitions  of  the  components  shown  in  Fig.  9, 
the  framework  generated  a  DataFusion  instance  to  detect 
human  activity  in  an  area  under  surveillance. 
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Fig.  9.  Components  with  associated  properties  to  be  used  in  a  suivcillanoe 
mission 


Fusion  of  data  from  a  profiling  sensor  and  a  classifier 
algorithm  is  accomplished  with  a  SPARQL7  query.  The 
algorithms  (Java  functions)  are  invoked  using  a  feature,  which 
is  part  of  the  SPARQL  library,  called  the  SPARQL  Inferencing 
Notation  (SPIN)8  developed  by  TopQuadrant  Inc.  Fig.  10 
shows  an  example  of  the  SPARQL  query  that  is  used  to 
generate  the  DataFusion  instance  that  combines  a  profiling 
sensor  data  output  to  the  output  of  a  visualizer.  The  adjoining 
object  diagram  illustrates  the  ontology  instantiation  in  the 
CONSTRUCT  query. 


In  the  query,  the  BIND  clause  in  the  last  highlighted  line 
contains  the  invocation  to  a  Java  class,  which  is  wrapped  as 
http  ://ksl.engr.  iupui ,  edu/re  source#  Silhouette  Vi  ewer  (aliased  as 
ksliures.  Silhouette  Viewer),  an  RDF  resource. 

The  coupling  between  sensor  system  output  and  algorithm 
inputs  is  based  upon  compatible  data  formats.  Datatype 
properties  such  as  hasAlgorithmlnputDataFormat  and 
hasSens or OutputDataF 'or mat  map  the  specific  component  to 
an  enumeration  of  string  literals  such  as  “TextJData”  or 
“Static_Image.”  To  combine  this  output  with  the  output  from  a 
classifier  algorithm,  the  query  in  Fig.  10  can  be  extended  to 
add  an  invocation  to  a  classifier  algorithm  as  shown  in  Fig.  11. 
The  new  invocation  has  been  highlighted  in  grey  in  the  figure. 
Note  the  new  component  that  has  been  added  to  the 
DataFusion  instance  in  the  object  diagram. 
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Fig.  11.  Extending  the  SPARQL  query  in  Fig.  11. 
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IV.  Discussion 

We  have  presented  a  simple  ontology-based  framework 
which  fuses  data  from  multiple  sources  enabling  situation 
awareness  for  a  human  operator.  There  are  two  important 
points  to  emphasize.  First,  the  ontological  framework  may 
facilitate  resource  control.  By  having  a  human  operator  verify 
the  human  activity  detected  by  the  sensor  systems  and 
algorithmic  processes,  appropriate  decisions  can  be  made  in 
response  to  the  sensed  event.  The  human  operator  helps  the 
framework  avoid  false  positives  from  the  autonomous 
algorithmic  process,  thus  managing  critical  resources 
optimally,  such  as  deploying  unmanned  aerial  vehicles  for 
further  reconnaissance.  Second,  the  ontological  framework 
produces  sensor  fusion  by  fusing  the  two  pieces  of  infonmation 
from  the  profile  visualizer  and  the  autonomous  classifier, 
which  is  more  valuable  to  a  human  operator  making  decisions 
than  just  having  access  to  one  piece  of  information.  Therefore, 
the  presented  framework  is  essentially  a  data-to-decision 
framework.  The  framework  can  also  be  extended  to 
accommodate  heterogeneous  sensors,  such  as  vibration  and 
acoustic  sensors  in  addition  to  the  profiling  sensors.  Fig.  12 
shows  an  extension  of  the  data  fusion  concept  introduced  in 
Figs.  5-8  to  mclude  outputs  from  heterogeneous  sensors,  such 
as  profiling  sensors  and  vibration  sensors,  in  combination  with 
different  classification  algorithms  to  provide  much  more 
information  about  the  situation.  Outputs  from  other  sensors, 
such  as  cameras  and  acoustic  sensors  (not  shown  here)  can  also 
be  combined  in  this  manner. 


Fig.  12.  Data  fusion  from  heterogeneous  sensors  results  in  more  evidence  being 
presented  to  a  human  agent 


In  terms  of  the  JDL  sensor  fusion  model,  the  ontological 
framework  performs  level  1  object  refinement  when  the 
DataFusion  instance  fuses  the  generated  profiles  to 
autonomous  classifications  Level  5  cognitive  refinement  (user 
refinement)  also  takes  place  when  the  synthesized  DataFusion 
instance  fuses  with  a  human  operator’s  decision  to  verify  the 
autonomous  classification  via  the  visualized  profile. 

V .  Conclusions  and  future  directions 

The  ontology -based  framework  presented  in  this  paper 
can  be  used  to  present  fused  information  obtained  from  various 


sources  to  a  human  observer  so  as  to  improve  confidence  in  the 
perceived  situation.  Sensor  systems  can  be  tasked,  algorithms 
can  be  dynamically  invoked  from  this  framework,  and  the 
output  data  can  be  assimilated  and  presented  to  a  human  agent 
enabling  situation  awareness  and  appropriate  decision  making. 

A  shortcoming  of  the  current  framework  is  the  necessity  to 
explicitly  specify  the  matching  criteria  for  the  sensor  systems 
in  the  SPARQL  query.  Note  the  SPARQL  query  example  in 
Fig.  10  invoking  the  silhouette  viewer  algorithm.  If  a  classifier 
algorithm  needed  to  be  integrated  and  invoked  as  well,  an  extra 
constraint  would  have  to  be  added  in  the  WHERE  clause  and  at 
least  two  other  unmatched  variables  would  need  to  be  specified 
in  the  query.  Given  the  inherent  nesting  of  the  sensor  systems 
in  the  generated  system,  we  are  working  on  an  interface  for 
dynamically  generating  the  SPARQL  query  from  the 
specifications  of  the  selected  components.  This  interface  would 
be  the  backend  for  a  GUI-  based  application  where  each  of  the 
components  would  be  displayed  on  a  selection  panel 

We  are  also  investigating  the  use  of  REST  services  [27-28] 
to  encapsulate  the  sensor  systems  and  algorithms  in  a  service- 
oriented  architecture  (SO A).  Each  component  would  be 
exposed  as  an  endpoint  interface  with  a  specific  URL  The 
specifications  of  each  component  would  be  encoded  in  a 
Semantic  Web  compatible  format  to  enable  discovery, 
invocation,  and  dynamic  composition  on  an  intranet  or  even 
the  secure  Internet. 

In  the  future,  the  fusion  architecture  is  to  be  expanded  to 
accommodate  multiple  profiling  sensors  in  a  given  area.  For 
example,  there  may  be  M  profiling  sensors  along  a  path  and  N 
different  classification  algorithms.  The  fusion  architecture 
could  be  tasked  to  produce  an  output  that  combines  and 
appropriately  summarizes  the  outputs  of  all  these  components 
for  an  end  user,  as  shown  in  Fig.  13.  The  next  logical  step 
would  be  to  use  heterogeneous  sensors,  such  as  acoustic  and 
vibration  sensors  and  their  corresponding  classifying 
algorithms,  to  enable  situation  awareness. 
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Fig.  13.  Data  fusion  combining  output  from  heterogeneous  sensors  with 
multiple  algorithms 
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ABSTRACT 

A  profiling  sensor  has  been  realized  using  a  vertical  column  of  sparse  detectors  with  the  sensor’s  optical  axis  configured 
perpendicular  to  the  plane  of  the  vertical  column  of  detectors.  Traditionally,  detectors  of  the  profiling  sensor  are  placed 
m  a  sparse  vertical  column  configuration.  A  subset  of  the  detectors  may  be  removed  from  the  vertical  column  and  placed 
at  arbitrary  locations  along  the  anticipated  path  of  the  objects  of  interest,  forming  a  custom  detector  array  configuration. 
Objects  passing  through  the  profiling  sensor’s  field  of  view  have  traditionally  been  classified  via  algorithms  processed 
off-line.  However,  reconstruction  of  the  object  profile  is  impossible  unless  the  detectors  are  placed  at  a  known  location 
relative  to  each  other.  Measuring  these  detector  locations  relative  to  each  other  can  be  particularly  time  consuming, 
making  this  process  impractical  for  custom  detector  configuration  in  the  field.  This  paper  describes  a  method  that  can  be 
used  to  determine  a  detector’s  relative  location  to  other  detectors  by  passing  a  known  profile  through  the  sensor’s  field  of 
view  as  part  of  the  configuration  process.  Real-time  classification  results  produced  by  the  embedded  controller  for  a 
variety  of  objects  of  interest  are  also  described  in  the  paper. 

Keywords:  Persistent  surveillance  sensor,  profiling  sensors,  embedded  controller,  classification,  object  detection, 
customizable  sensor  arrangement 


1.  INTRODUCTION 

Profiling  sensors  encompass  a  family  of  sensor  devices  and  configurations  that  are  used  to  acquire  profiles  or  silhouette 
data  of  objects  of  interest  for  classification  purposes1.  Variations  include  employing  an  array  of  active  near-IR  trip 
sensors,  a  passive  pyroelectric  approach2,3,  as  well  as  extracting  a  subset  of  pixels  from  the  image  of  conventional 
imagers  to  emulate  a  sparse  detector  array  5‘6.  Deployment  scenarios  for  such  profile  sensors  include  border  monitoring 
over  open  terrain  for  persistent  surveillance.  Profiling  sensors  must  be  able  to  automatically  distinguish  between  humans 
and  non-human  objects,  such  as  deer,  rabbits,  horses  and  other  animals  that  reside  in  the  area  of  interest.  Objects  are 
desired  to  be  classified  as  either  human,  vehicle,  or  animal. 

The  profile  sensor  considered  for  implementation  m  this  paper  utilizes  a  vertically  oriented  column  of  sixteen  near-IR 
optical  trip  wires  (OTWs).  These  detectors  are  arranged  approximately  5  inches  apart  from  one  another,  forming  a 
sparse  array  with  their  optical  axes  configured  perpendicular  to  the  plane  of  the  vertical  column  as  shown  in  Figure  1  (a). 
Each  detector  is  coupled  with  its  own  reflector  mounted  on  an  opposing  platform7  8,9.  Objects  that  pass  through  the 
beams  record  the  ‘on’  or  ‘off  state  for  each  detector  for  a  particular  time  sample,  which  is  then  recorded.  The  object 
profile  is  generated  by  collecting  the  collection  of  time  samples  m  which  a  detector  break-beam  triggering  event 
occurred,  forming  a  silhouette  of  the  object  that  passed  through  the  beams  of  the  array  as  shown  in  Figure  1  (b).  The 
active  IR  sensing  elements  require  that  a  subject  pass  through  the  beams  to  be  detected.  This  requirement  limits  the 
OTW  to  deployments  in  which  objects  of  interest  must  pass  through  a  very  constricted  area. 
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Detectors  of  the  profiling  sensor  are  traditionally  placed  in  a  sparse  vertical  column  configuration.  Since  no  horizontal 
spacing  exists  between  the  sensing  elements,  profiles  generated  by  the  vertical  column  are  constructed  by  placing  the 
recorded  time  samples  next  to  each  other. 


(a)  (b) 

Figure  1  (a)  Traditional  profile  sensor  with  vertically  oriented  detectors  and  (b)  example  profile  of  a  horse  generated  by  a 
profile  sensor  with  vertically  oriented  detectors 


Requiring  a  subject  to  pass  through  the  sensor  to  obtain  its  profile  presents  a  limitation  that  is  easy  to  counter.  If  a 
person  recognizes  the  sensor  that  is  being  used  to  monitor  him/her,  he/she  merely  needs  to  walk  around  the  sensor  to 
avoid  detection.  Good  concealment  is  therefore  essential.  Part  of  the  concealment  may  be  done  by  breaking  the  vertical 
array  apart,  and  distributing  the  detectors  along  an  anticipated  path  of  objects  of  interest;  thereby,  reducing  the  obtrusive 
size  of  the  single  array  pole  and  its  reflector8  10.  An  alternative  profiling  sensor  prototype  design  is  shown  in  Figure  2. 
Note  that  this  particular  configuration  is  a  prototype  with  horizontal  distances  of  approximately  15-35  cm  between 
detectors.  Actual  field  implementation  would  place  individual  detectors  where  maximum  concealment  can  be  achieved, 
possibly  with  significantly  greater  horizontal  separation. 


Figure  2:  Customized  sensor  array  prototype 


Acquiring  a  profile  from  the  detector  elements  that  are  not  in  the  same  plane  requires  that  the  precise  locations  of  the 
detector  elements  be  known  to  synchronize  timing  between  them.  Measuring  these  distances  by  conventional  techniques 
is  particularly  time-consuming,  especially  if  the  detectors  are  to  be  placed  at  significant  horizontal  distances  from  each 
other.  Long  setup/measurement  time  makes  the  custom  configurations  impractical  for  deployment  in  the  field,  especially 
when  deployment  time  is  minimal.  A  quick  method  of  measuring  the  distance  between  the  detectors  is  therefore 
warranted. 


2.  SIMPLE  CLASSIFICATION  TECfflNQUE  USING  CUSTOM  ARRAY 

A  simple  classifier  can  be  implemented  using  the  custom  array  by  summing  the  ‘on*  and  'off  states  of  the  detector  along 
each  row  as  an  object  passes  through  the  sensor.  The  summation  data  along  each  row  is  treated  as  an  independent 
feature  and  compared  to  training  data  sets  of  human,  vehicle,  or  animal.  The  object  is  subsequently  classified  using 
Naive  Bayesian  classifier  A  leave-one-out  classification  study  against  the  profile  sensor’s  acquired  sample  library10 
revealed  that  this  technique  can  obtain  a  92%  classification  rate  using  Mahalanobis  distance.  Its  ease  of  implementation 
makes  it  particularly  appealing  for  a  low-resource  microcontroller.  However,  the  technique  fails  to  calculate  the  object’s 
velocity  or  its  direction  of  travel,  both  of  which  may  provide  valuable  information  on  a  passing  subject.  Higher-yielding 
classification  algorithms  have  been  developed  for  profiles  generated  by  the  sensor  with  vertically  oriented  detectors7,8,11. 
However,  to  utilize  these  algorithms,  the  profile  must  be  reconstructed  to  appear  as  if  it  were  generated  by  the  sensor 
with  vertically  oriented  detectors. 

3.  CUSTOM  ARRAY  PROFILE  RECONSTRUCTION  TECHNIQUE 

Detector  timing  for  the  vertical  column  configuration  is  synchronized  by  the  placement  of  the  detectors  along  the  same 
column,  as  illustrated  by  Figure  3.  Most  subjects  passing  through  the  sensor’s  field  of  view  do  not  have  a  flat  leading 
edge;  therefore,  their  profiles  will  illustrate  a  non-straight  edge.  Passing  a  straight,  vertical  object  through  the  array  will 
cause  all  16  detectors  to  trip  simultaneously,  as  illustrated  by  Figure  4. 


Figure  3:  Subject  passing  through  a  vertical  column  sensor  Figure  4:  Vertically  straight  object  passing  through  vertical  array 

Removing  these  detectors  from  the  same  column  destroys  the  synchronization  between  them.  Passing  the  straight 
vertical  object  through  an  array  in  which  the  detectors  are  not  vertically  co-located  will  create  a  profile  in  which  the 
detector  elements  ‘on’  and  ‘off  times  are  modified  by  the  placement  of  the  detectors.  Figure  5  illustrates  the  passing  of 
a  vertically  straight  object  through  a  modified  array.  Passing  this  rectangular  ‘calibration  rectangle’  through  the  sensor  is 
done  as  part  of  a  calibration  routine  to  define  the  distances  between  each  detector  pair.  Since  it  is  known  that  the  leading 
edge  of  this  rectangular  object  is  vertically  straight,  the  physical  location  of  the  detectors  can  be  determined  by  counting 
file  number  of  time  samples  between  a  detector  pair’s  first  transition  state.  For  example,  if  there  are  X  time  samples 


between  detectors  on  row  i  and  row  j,  then  it  can  be  assumed  that  the  physical  distance  between  detectors  on  rows  i  and  j 
for  subsequent  samples  should  be  offset  by  the  same  X  number  of  time  samples.  The  number  of  samples  between  rows 
is  found  by  searching  for  the  first  transition  state  of  each  row  from  ‘on*  to  ‘off,  corresponding  to  the  leading  edge  of  the 
passing  rectangle.  Similarly,  the  trailing  edge  may  also  be  used  if  the  calibrating  vertical  object  has  a  vertical  rear  edge. 
Note  that  the  vertical  calibration  rectangle  must  be  passed  through  the  array  at  a  relatively  constant  speed  to  trip  the 
detectors  at  their  correct  physical  locations. 
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Figure  5:  Vertically  straight  object  passing  through  custom  array 


These  time  differences  (corresponding  to  the  positions  of  the  individual  detectors)  are  then  subtracted  from  the  raw 
timing  profile  of  each  subsequent  subject  passing  through  the  array  to  rebuild  the  proper  profile.  Figure  6  illustrates  a 
subject  passing  through  the  prototype  custom  array,  raw  profile  generation  and  the  subsequent  reconstruction  of  the  raw 
profile  data  to  form  a  properly  oriented  profile. 
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Figure  6:  Subject  profile  reconstruction  after  passing  through  custom  array 


4  IMPLEMENTATION 

Experimentation  for  the  custom  array  was  realized  by  placing  the  detector  locations  at  specified  heights  and  random 
locations.  Detectors  are  placed  at  random  horizontal  locations  on  rows  that  are  vertically  separated  5”  from  each  other, 
consistent  with  the  5”  separation  between  detectors  of  the  vertical  column  array  profile  sensor.  This  implementation  was 
realized  by  modifying  the  single-column  array.  The  single-column  array  is  constructed  of  PVC  pipe  sections  and  is 


designed  for  rapid  field  data  collection  deployment.  Detectors  of  the  single  column  array  are  paired  mto  two-element 
sections  for  ease  of  alignment  and  to  minimize  the  number  of  necessary  electrical  connections.  The  eight  detector  pairs 
are  removed  from  the  column  array  and  placed  on  a  section  of  PVC  pipe  to  set  them  at  their  original  vertical  heights. 
Horizontal  spacing  is  random. 


Figure  7:  Custom  sensor  array  prototype  realized  on  PVC  pipe. 

Note  that  the  detector  arrangement  in  Figure  7  shows  the  detectors  in  a  roughly  diagonal  pattern.  Detector  placements  do 
not  necessarily  require  such  an  arrangement,  but  may  be  arranged  in  any  fashion.  This  particular  configuration  was 
merely  chosen  as  an  illustration  for  this  paper  and  to  construct  the  prototype  in  Figure  7.  Also  note  that  the  element  pairs 
do  not  necessarily  have  to  lie  on  the  same  vertical  plane.  Detectors  in  Figure  7  were  left  paired  together,  due  to  their 
construction  Figure  8  (a)  illustrates  a  human  carrying  a  2x4  vertically  through  the  sensor  (moving  from  left  to  right 
through  the  sensor  in  Figure  7)  as  part  of  a  calibration  routine.  The  calibration  routine  determines  the  first  transition  state 
of  each  detector  and  finds  the  time  difference  between  the  profile's  first  detector's  trigger  event  and  the  first  trigger  event 
along  each  row.  The  profile  is  ‘straightened'  by  subtracting  these  time  differences  along  their  respective  rows,  as  shown 
in  Figure  8.  Note  that  the  same  2x4  beam  is  shown  again  in  Figure  8  (b)  in  a  more  recognizable  format. 


Figure  8:  Person  carrying  a  vertical  object  through  the  custom  sensor  array  of  Figure  7;  (a)  uncorrected  and  (b)  corrected. 
Vertical  beam  is  shown  in  oval. 


Figure  9:  Reconstructed  human  profile  collected  with  custom  array 


Subsequent  data  acquisitions  have  this  same  row  shifting  routine  performed  on  them.  Figure  9  shows  the  corrected 
profile  of  a  person  passing  through  the  array  without  the  2x4  used  for  detector  timing  calibration.  This  technique  of 
shifting  the  start  position  of  each  of  the  rows  appears  to  account  for  the  random  horizontal  placement  of  detectors. 
Although  the  array  may  be  placed  in  any  customized  configuration,  the  specific  detector  locations  must  remain  fixed 
after  the  calibration  routine  is  executed.  If  the  array  is  modified,  the  calibration  routine  must  be  performed  again  to  find 
the  detectors’  positions  relative  to  one  another. 


Note  that  data  collected  from  the  vertical  column  array  profile  sensor  consists  of  timing  samples.  A  slowly  travelling 
subject  will  generate  a  profile  of  greater  width  than  if  it  were  to  pass  through  the  sensor  quickly.  Thus,  the  number  of 
time  samples  used  to  generate  a  profile  alone  is  not  an  accurate  measurement  tool  to  indicate  the  physical  width  of  the 
passing  subject.  Merely  applying  the  time  sample  shifting  to  each  of  the  rows  is  not  an  entirely  accurate  method  of 
realigning  a  profile  since  subjects  may  pass  through  the  array  at  speeds  other  than  which  the  detector  distances  were 
calibrated.  Since  t  =  d/vt  the  amount  of  shift  /  applied  to  each  row  is  a  function  of  the  detector  element  distances  d  and 
the  velocity  v  of  the  travelling  subject,  not  merely  a  count  of  the  number  of  time  samples  between  rows.  Subjects  with  a 
speed  v,  which  may  be  different  than  that  of  objects  used  in  the  calibration,  will,  therefore,  have  malformed  rebuild 
profiles.  Furthermore,  correcting  the  profiles  due  to  time  only  does  not  account  for  the  direction  of  travel  that  a  subject 
may  take  through  the  sensor.  Travelling  in  the  direction  opposite  from  what  the  array  was  calibrated  causes  the  row  data 
to  be  shifted  in  the  opposite  direction  as  shown  in  Figure  1 0  (b).  Although  this  technique  of  profile  rebuilding  may  work 
appropriately  if  we  make  the  assumption  that  subjects  will  always  pass  at  the  same  speed,  it  fails  when  the  assumption  is 
violated.  Therefore,  a  more  robust  calculation  technique  is  required. 


(a)  (b) 

Figure  10;  Incorrect  ‘corrected’  arrays  caused  by  (a)  speed  other  than  calibration  speed  and  (b)  travelling  opposite  direction 
from  calibration  route. 


5.  VELOCITY  CALCULATION  AND  INCORPORATION 


A  more  accurate  method  of  correctly  rebuilding  the  profiles  requires  that  the  physical  distances  between  detectors  and 
the  actual  velocity  of  the  passing  subject  be  determined.  Using  these  parameters,  a  more  accurate  time  shifting  value 
may  be  calculated  for  each  detector  state.  Timing  differences  between  the  detectors  are  directly  proportional  to  the 
horizontal  distances  between  them  and  are  determined  by  passing  a  vertically  oriented  object  through  the  array  as 
described  earlier.  However,  the  distance  between  any  one  pair  of  detector  elements  must  be  known.  Ratios  of  the 
numbers  of  tune  samples  recorded  between  element  pairs  are  computed.  Assuming  that  the  calibration  rectangle  is 
passed  through  the  sensor  at  a  relatively  constant  speed,  the  physical  distances  are  a  product  of  the  time  ratios  between 
the  detectors  and  the  known  horizontal  distance  between  the  two  designated  detectors.  These  distances  are  calculated 
several  times  as  part  of  the  calibration  routine  and  averaged  along  each  row. 

A  subject’s  velocity  through  the  profile  sensor  is  calculated  by  averaging  the  individual  velocities  v,y  generated  between 
every  possible  detector  pair  along  the  leading  and  trailing  edges  of  the  profile,  given  by  the  expression 


mean  velocity  = 
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where  i  ^  j,  and  M  =  the  uppermost  detector  trigger  event3.  Velocity  between  each  detector  pair  is  defined  as 
=  dij  /  ty  where  dy  is  the  physical  distance  between  detectors  i  and  j,  and  ty  is  the  number  of  samples  between  the 
detectors  i  and  j  to  the  leading  or  trailing  edge  of  the  profile.  Although  seemingly  computationally  intense,  the  number 
of  detector  pair  velocity  calculations  is  limited  to  M*(M-1)  /  2,  where  the  maximum  value  of  M  is  the  number  of 
detectors  in  the  array;  a  maximum  of  only  1 20  possible  calculations  along  each  edge  is  possible  for  a  1 6  detector  array. 
Combinations  where  t=0  cause  division  by  zero  and  are  not  considered.  The  implementation  of  the  vertically  paired 
sensor  detectors  causes  horizontal  pairings  to  have  a  time  difference  of  zero,  thereby  eliminating  those  particular  detector 
pair  velocity  calculations  from  consideration  and  further  reducing  the  computational  requirements  of  the  microcontroller. 

A  rectangular- shaped  object  passing  through  the  sensor  would  generate  constant  detector  pair  velocities.  Upright, 
walking  humans  are  mostly  rectangular  in  shape,  generatmg  similar  velocities  along  the  horizontal  detector  pairings  for 
the  front  and  rear  edges  of  the  profile.  However,  not  all  profiles  share  this  rectangular  pattern.  The  overhanging  head  of 
an  animal,  sloped  vehicle  windshields,  swinging  arms,  etc.,  caused  by  premature  detector  triggering,  can  generate 
abnormal  detector  pair  velocity  calculations.  These  particular  velocities  are  eliminated  by  setting  a  threshold  of  a 
maximum  reasonable  velocity  for  individual  calculated  velocity  values  v,^. 

Once  an  overall  subject  velocity  is  calculated,  the  amount  of  time  required  to  shift  each  row  ty  is  calculated  in  samples  by 
ty  a  dij  l  v,  where  dij  is  the  physical  distance  between  the  detector  elements  i  and  j  found  from  the  calibration  routine. 

Note  that  this  velocity  calculation  is  capable  of  generating  both  positive  and  negative  values;  with  this  data  a  direction  of 
travel  through  the  gates  can  be  determined.  Figure  1 1  illustrates  the  more  properly  reconstructed  data  of  the  same 
profiles  found  in  Figure  10  by  this  more  accurate  technique. 


Figure  1 1:  more  accurately  reconstructed  profiles  of  Figure  10  generated  by  velocity  incorporation. 


6.  RESULTS 


The  low  resource  microcontroller  has  proven  to  be  effective  at  rebuilding  profiles  from  a  custom  array  when  utilizing  the 
passing  object’s  velocity.  Figure  1 2  illustrates  the  raw  and  reconstructed  profiles  for  several  objects  passing  through  the 
custom  array. 
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Figure  12:  Raw  and  reconstructed  profiles  collected  by  custom  array 


Incorporation  of  velocity  allows  a  more  accurate  profile  width  to  be  determined  from  data  acquired  from  profiling 
sensors3.  Classification  algorithms,  such  as  the  height/width  ratio  and  six- feature  technique,  are  based  strongly  on  the 
width  of  the  sample3,1  *.  Improving  the  determination  of  the  width  of  an  object  will  improve  the  overall  classification  of 
objects  by  profiling  sensors.  However,  to  date,  there  are  few  samples  within  our  library  of  profiles10  that  have  the 
physical  width  derived  based  on  the  subject’s  velocity.  This  “distance”  library  is  composed  of  profiles  of  30  humans,  32 
animals,  and  29  vehicles.  Comparison  of  a  distance-based  width  profile  to  the  timing-based  width  of  profiles  acquired 
by  the  vertical  array  profile  sensor  is  likely  to  lead  to  fallacious  results.  Take-one-out  classification  studies  were 
performed  on  the  data  within  our  library  utilizing  the  height- width  ratio  and  six-feature  techniques  with  54%  and  85% 
results,  respectively.  Classification  confusion  matrices  for  the  two  classification  techniques  are  displayed  in  Tables  1  a 
and  lb.  Incorporation  of  a  larger  training  set  is  hypothesized  to  increase  these  classification  results. 


Table  la:  Height/Width  Feature  Confustion  Matrix _  Table  lb:  Six-Feature  Classification  Confusion  Matnx 


True  Classification 

True  Classification 
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Feature 

Classification 

Human 

Animal 
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Human 

Animal 

Vehicle 

Human 
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1 
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Human 
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4 

0 

Animal 
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5 
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Classification 
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The  simple  technique  of  summing  the  detector  ‘off  states  is  currently  utilized  as  a  classification  tool  on  the 
microcontroller  until  future  data  collection  events  can  be  performed  with  the  custom  array.  Detector  state  events  are 
summed  and  normalized  by  the  highest  number  of  detector  events  of  the  sixteen  row  features,  making  this  technique 
particularly  appealing:  the  “timing”  library  may  be  compared  against  the  raw  timing  features  of  a  subject  test  case.  The 
microcontroller  has  yielded  a  real-time  85%  classification  rate  using  this  feature  for  a  wide  variety  of  humans,  vehicles, 
and  animals  during  field  data  collection  events.  Classifications  are  made  by  comparing  the  normalized  energy  features 
along  each  row  with  a  Na'ive  Bayesian  classifier  trained  against  the  ‘timing’  library.  These  real-time  classification 
results  are  displayed  in  the  confusion  matrix  of  Table  2: 


Table  2:  Real-time  Row  Energy  Feature  Classification  Confusion  Matrix 


True  Classification 

Real-time  Row  Energy 
Feature  Classification 

Human  (30) 

Animal  (32) 
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Human 
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32 
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Vehicle 

1 

7 

21 

Implementation  of  the  calibration  routine,  data  collection,  velocity  calculation,  profile  reconstruction,  and  classification 
algorithm  has  been  completed  on  a  Rabbit  BL4S200  microcontroller.  Thus  far,  the  microcontroller  has  shown  promising 
results  when  coupled  to  the  near-IR  profiling  sensor  array  in  both  vertical  and  custom  detector  configurations. 

7.  CONCLUSIONS 

Previous  work  has  shown  the  feasibility  of  employing  the  family  of  profiling  sensors  to  detect  objects  of  interest  for  wide 
area  surveillance.  The  focus  of  this  paper  is  to  demonstrate  the  feasibility  of  implementing  a  custom  array  of  random 
detector  placements  with  minimal  setup  effort  Efforts  show  that  the  system  is  capable  of  an  85%  classification  rate 
using  mutually  exclusive  features  on  a  sample  data  set.  Future  work  involves  the  collection  of  additional  samples  and 
more  rigorous  testing  of  the  custom  array  configuration  against  a  more  dynamic  test  subject  set. 
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Abstract:  The  deployment  of  ubiquitous  sensor  systems  and  algorithms  has  led  to  many 
challenges,  such  as  matching  sensor  systems  to  compatible  algorithms  which  are  capable  of 
satisfying  a  task.  Compounding  the  challenges  is  the  lack  of  the  requisite  knowledge 
models  needed  to  discover  sensors  and  algorithms  and  to  subsequently  integrate  their 
capabilities  to  satisfy  a  specific  task.  A  novel  ontological  problem-solving  framework  has 
been  designed  to  match  sensors  to  compatible  algorithms  to  form  synthesized  systems, 
which  are  capable  of  satisfying  a  task  and  then  assigning  the  synthesized  systems  to 
high-level  missions.  The  approach  designed  for  the  ontological  problem-solving 
framework  has  been  instantiated  in  the  context  of  a  persistence  surveillance  prototype 
environment,  which  includes  profiling  sensor  systems  and  algorithms  to  demonstrate 
proof-of-concept  principles.  Even  though  the  problem-solving  approach  was  instantiated 
with  profiling  sensor  systems  and  algorithms,  the  ontological  framework  may  be  useful 
with  other  heterogeneous  sensing-system  environments. 

Keywords:  sensor  networks;  sensor  ontology;  profiling  sensors;  ontological  framework 
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1.  Introduction 

Dynamically  matching  sensor  systems  to  algorithms  to  satisfy  a  task  poses  a  significant  challenge  in 
sensor  networks.  The  challenge  is  made  even  more  difficult  because  sensor  systems  and  algorithms  are 
not  typically  designed  independently,  which  often  limits  their  reuse  in  tasks  that  may  not  have  been 
anticipated  when  the  sensors  and  algorithms  were  first  deployed.  Compounding  the  challenge  is  the 
lack  of  knowledge  and  data  models,  which  descnbe  sensor  and  algorithm  capabilities,  properties,  and 
relationships  [1-6].  The  focus  of  this  paper  is  on  the  reasoning  process  used  in  a  novel  ontological 
problem-solving  framework,  which  can  be  leveraged  by  software  agents  on  sensor  networks,  to 
opportunistically  match  sensor  systems  to  independently  designed  algorithms  to  form  synthesized 
systems  capable  of  satisfying  a  task. 

1 .1 .  Ontological  Problem-Solving  Framework 

The  ontological  problem-solving  framework  (Figure  1)  has  the  overall  goal  to  discover  and  match 
sensor  systems  to  compatible  algorithms  to  form  a  synthesized  system,  which  is  capable  of  satisfying  a 
given  subtask.  The  synthesized  systems  and  other  algorithms  may  then  be  matched  to  form  more 
complex  synthesized  systems,  which  may  then  be  assigned  to  tasks  of  high-level  missions  (Figure  2). 
The  ontological  problem-solving  framework  will  then  coordinate  all  matched  and  synthesized  sensor 
systems  and  algorithms  to  complete  the  missions.  The  problem-solving  approach  could  have  been 
developed  with  standard  database  technologies  and  SQL  queries.  However,  one  of  the  issues  that 
makes  discovering  and  matching  sensors  to  algorithms  problematic  is  the  lack  of  knowledge  models 
used  to  describe  those  systems. 

Figure  1.  Overview  of  ontological  problem-solving  framework. 
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Figure  2.  Creation  of  synthesized  systems  which  are  then  assigned  to  subtasks  of 
high-level  missions  via  the  ontological  problem-solving  framework. 
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The  knowledge  models  also  need  to  leverage  well-defined  semantics  in  a  machine-interpretable 
format  so  other  agents  may  interact  with  the  described  systems.  The  requirement  to  opportunistically 
match  sensors  to  algorithms  increased  the  need  to.  use  ontologies  (which  specify  the  semantics)  and 
rules  based  on  description  logic  to  infer  which  components  may  be  used  to  form  synthesized  systems. 
The  knowledge  models  used  by  the  ontological  problem-solving  framework  may  then  be  leveraged  by 
other  systems  for  more  complex  inference  if  needed.  The  ontological  problem-solving  framework  was 
developed  using  the  TopBraid  Maestro  software  by  TopQuardrant  [7],  which  uses  the  web  ontology 
language  (OWL)  [1-6]  for  knowledge  capture,  SPARQL  [8]  for  specifying  rules,  and  the  TopSpin 
inference  engine  for  interpreting  the  rules.  Other  systems,  such  as  Protege,  which  uses  JESS  and 
SWRL  [1-6],  could  have  also  been  used  to  develop  the  ontological  problem-solving  framework.  The 
main  focus  of  this  paper  is  to  detail  the  reasoning  process  the  ontological  problem-solving  framework 
uses  to  match  sensor  systems  to  compatible  algorithms  to  form  synthesized  systems,  which  are  capable 
of  satisfying  a  given  task. 

1 2.  Matching  Sensors  to  Algorithms 

Engineers  often  design  an  algorithm  for  a  specific  sensor  system.  This  dependence  makes  the 
algorithm  difficult  to  use  with  other  sensors  opportunistically  based  on  ever-changing  persistence 
surveillance  goals.  If  sensors  and  algorithms  are  designed  independently,  then,  a  problem-solving 
approach  must  enable  the  matching  of  a  sensor  to  a  compatible  algorithm  to  achieve  a  task,  such  as 
formatting  the  sensor  data  for  a  specific  purpose  or  extracting  pixels  from  an  imaging  sensor  for 
subsequent  processing.  The  composition  of  matched  sensor  systems  and  compatible  algorithms  to 
achieve  a  task  can  be  made  even  more  difficult  if  an  algorithm  requires  multiple  data  sources 
(Figure  3(a)),  or  if  a  chain  of  multiple  sensors  and  algorithms  must  be  composed  to  achieve  subtasks 
supporting  an  overall  task  (Figure  3(b)).  The  problem-solving  approach  must  describe  the  relationship 
between  the  preconditions  and  post  conditions  of  the  algorithms,  as  well  as  descriptions  of  the  raw 
data,  and  possibly  features  generated  by  the  sensor  systems  [9-12]. 
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Figure  3.  (a)  Algorithm,  which  requires  data  from  two  sensor  systems,  matched  to  two 
compatible  sensor  systems,  (b)  Algorithm  matched  to  a  compatible  algorithm,  which  is 
also  matched  to  a  compatible  sensor  system. 
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1 .3.  Related  Work 

There  have  been  several  approaches  and  tools  developed  to  address  in  part  the  challenge  of 
matching  sensors  to  compatible  algorithms.  These  techniques  and  tools  include,  but  are  not  limited  to, 
Sensor  Fabric  [9,13-15],  Sensor  OASiS  [16],  Agilla  [17-19],  Semantic  Streams  and  SONGS  [20,21], 
and  CIEDETS  [22,23],  Other  research  efforts  focused  on  the  development  of  ontologies  that  describe 
sensors  and  their  respective  capabilities,  such  as  OntoSensor  [2-6],  Sensor  Network  Data  Ontology  [24], 
Sensor  and  Data  Wrapping  Ontology  [25],  Stimulus-Sensor-Observation  Ontology  [26],  Sensor 
Observation  and  Measurement  Ontology  [27],  Semantic  Sensor  Network  Ontology  [28],  Disaster 
Management  Sensor  Ontology  [29],  and  a  survey  of  sensor  ontologies  [30]  are  also  efforts  relevant  to 
our  work.  Other  work  promotes  a  logical  model  to  follow  while  developing  a  problem-solving 
approach.  For  example,  Sensor  Modeling  Language  (SensorML)  [31]  describes  high-level  conceptual 
models  using  Unified  Modeling  Language  (UML)  of  sensors,  algorithms,  and  supporting  notions  to 
facilitate  interoperability.  The  Open  Geospatial  Consortium  (OGC)  specify  draft  interoperability 
interface  standards  and  metadata  encodings  that  integrate  sensor  systems  into  information 
infrastructures,  such  as  Observations  and  Measurements  (O&M)  [32,33],  SensorML  [34],  Transducer 
Model  Language  (TML)  [35],  Sensor  Observation  Service  (SOS)  [36],  Sensor  Planning  Service 
(SPS)  [37],  Sensor  Alert  Service  (SAS)  [38],  and  Web  Notification  Services  (WNS)  [39],  Semantic 
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Streams  and  OntoSensor  are  two  important  efforts  because  of  their  use  of  semantics  and  ontologies. 
Semantic  Streams  and  the  follow  up  SONGS  effort  were  developed  by  Microsoft  to  facilitate  queries 
to  determine  capabilities  and  subsequent  tasking  of  sensors  and  algorithms  Semantic  Streams  uses 
event  streams,  which  are  collected  raw  data  from  sensor  systems  with  meta  information  attached,  and 
inference  units,  which  operate  on  event  streams  by  creating  semantic  information  about  the  event 
streams.  Queries  posted  to  Semantic  Streams  are  broken  down  into  one  or  more  of  the  inference  units 
(Figure  4).  SONGS  adds  the  use  of  an  ontology  to  describe  the  inference  units.  Instead  of  queries  being 
directly  mapped  to  inference  units,  the  approach  can  infer  which  inference  units  may  satisfy  a  given 
query  [20,21].  OntoSensor  is  a  semantic-web-compatible  ontology  that  captures  knowledge  about 
sensor  systems  (Figure  5(a)).  OntoSensor  can  be  used  to  create  relationships  to  other  sensor  instances 
and  to  derive  properties  about  sensor  systems.  Software  agents  can  query  the  sensor  instance  data  to 
determine  the  capabilities  of  connected  sensor  systems.  Once  the  capabilities  of  the  sensor  systems 
have  been  detemiined,  other  agents  may  task  the  sensor  systems,  for  example,  retrieving  humidity  data 
for  a  specified  time  period  (Figure  5(b))  [2-6]. 

Figure  4.  Semantic  Streams  query. 


Figure  5.  (a)  Excerpt  of  the  OntoSensor  ontology,  (b)  Problem-solving  for  discovering  and 
tasking  sensor  systems  using  OntoSensor. 
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1  A.  Profiling  Sensor  Systems  and  Algorithms 

To  show  how  an  ontological  problem-solving  framework  can  address  the  challenge  of  matching 
sensor  systems  to  compatible  algorithms  for  a  specific  task,  a  family  of  unattended  ground  profiling 
sensors  (denoted  as  PFx,  in  which  PFi,  PF2,  PFn  are  different  types  of  profiling  sensors)  and 
algorithms  were  deployed  in  a  prototype  environment.  PFx  sensor  systems  provide  unique 
opportunities  for  dynamic  feature  extraction  through  extendable  algorithms  and  subsequent  tasking. 
The  main  purpose  of  PFx  sensors  is  to  capture  profiles  of  objects,  which  can  be  subsequently  classified 
by  algorithms  using  a  variety  of  techniques,  such  as  Naive  Bayes  algorithms,  neural  networks,  or 
support  vector  machines.  A  common  theme  of  all  PFx  sensors  is  that  they  are  intended  to  be  low  cost 
and  provide  a  profile  that  can  be  reliably  classified.  There  are  many  different  types  of  PFx  sensors, 
which  exploit  various  technologies,  including  a  family  of  PFx  imaging  sensors,  which  use  a  sparse 
detector  array.  PFx  sensors  include,  but  are  not  limited  to,  novel  imaging  sensors  in  the  visible,  near 
infrared,  short-wave  infrared,  mid-wave  infrared,  and  long-wave  infrared  bands.  One  of  the  initial  and 
simplest  approaches  to  a  PFx  sensor  was  a  prototype  that  used  a  sparse,  vertical  array  of  detectors.  One 
configuration  was  on  a  vertical  pole,  as  shown  in  Figure  6(a),  while  other  configurations  may  include  a 
horizontal  displacement  among  the  detectors  as  shown  in  Figures  7(a)  and  8.  Other  algorithms  may 
format  or  compress  the  raw  sensor  data  produced  by  PFx  sensors,  as  shown  in  Figure  6(b),  or  generate 
profiles  into  formats  such  that  other  algorithms  can  subsequently  process  the  data,  as  shown  in 
Figure  7(b).  One  example  is  a  visualization  algorithm,  which  may  generate  a  silhouette  of  an  object  for 
presentation  to  a  human  evaluator  for  classification.  Other  algorithms  that  process  PFx  data  may 
classify  silhouettes  as  humans,  animals,  or  vehicles  [12,40-46], 


Figure  6.  (a)  Near-IR  PFx  sensor  with  detectors  vertically  deployed,  (b)  Output  from  an 
algorithm  that  formats  PFx  sensor  data,  (c)  Output  from  an  algorithm  that  produces  a 
silhouette  from  formatted  PFx  data. 
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Figure  7.  (a)  PFx  sensor  with  detectors  deployed  vertically  with  a  horizontal  displacement, 
(b)  PFx  raw  data  formatted  by  an  algorithm  as  a  profile. 
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Figure  8.  (a)  PFx  sensor  with  detectors  deployed  vertically  with  a  specific  horizontal 
displacement,  (b)  PFx  sparse  detector  with  random  detector  displacement,  (c)  PFx  sparse 
detector  with  only  horizontal  displacement. 
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Tire  PFx  systems,  with  their  various  capabilities  and  relationships,  represented  a  unique  opportunity 
for  integration  onto  the  ontological  problem-solving  framework  (Figure  9).  The  following  section 
describes  in  detail  tire  novel  ontological  problem-solving  framework  using  PFx  sensors  and  algorithms 
to  illustrate  the  matching  of  sensor  systems  to  independently  designed  algorithms  for  a  task.  The 
problem-solving  approach  will  illustrate  how  PFx  sensors  are  matched  to  compatible  algorithms  for 
pixel  extraction,  profile  generation,  visualization,  and  various  other  tasks.  Even  though  the  PFx  sensors 
and  algorithms  are  used  for  proof-of-principle  aspects  of  the  ontological  problem-solving  framework, 
the  same  approach  may  be  extended  for  use  by  other  types  of  sensors  and  algorithms  to  achieve 
different  tasks. 
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Figure  9.  (a)  Representative  algorithm  types,  including  classifiers,  visualizers,  and  silhouette 
generators,  (b)  Representative  PFx  sensor  types,  including  sparse  detectors  and  imagers. 


2.  Reasoning  Process  to  Match  Sensor  Systems  to  Algorithms 

The  ontological  problem-solving  framework  uses  a  reasoning  process  that  leverages  knowledge 
management  techniques,  such  as  semantic  data  modeling  w;ith  ontologies,  to  address  the  challenge  of 
matching  sensors  to  compatible  algorithms  to  fonn  synthesized  systems  capable  of  satisfying  a  task. 
For  this  paper,  the  following  definitions  are  used  to  describe  sensors  and  algorithms.  A  sensor  is  a 
device  that  produces  raw  data  while  an  algorithm  uses  the  raw  data  for  further  processing  These 
definitions  are  similar  to  ones  put  forth  by  the  Open  Geospatial  Consortium,  such  as  defining  sensors 
as  processes  and  defining  sensors  and  algorithms  as  services  in  SensorML  [34],  Of  note  is  that 
low-level  algorithms,  which  may  reside  on  the  sensor  hardware,  are  now  considered  as  algorithms, 
which  are  not  part  of  tire  physical  sensor.  The  low-level  algorithms  may  be  device  drivers  or  software 
to  process  tire  raw  sensor  data  into  a  specific  format.  Separating  the  low-level  algorithms  from  the 
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specific  sensor  systems  facilitates  a  more  flexible  knowledge  representation  of  the  sensor  systems  and 
algorithms.  With  these  definitions,  meta-data,  such  as  sensor  and  algorithm  properties,  network 
communications,  data  fonnats,  etc.,  must  be  captured  to  explicitly  represent  the  relationships  among 
sensors  and  algorithms.  The  use  of  models  to  capture  knowledge  about  sensors  and  algorithms 
facilitates  inference  with  rules  based  on  description  logic.  The  knowledge  models,  rules,  and  inference 
engine  may  then  allow  other  agents  using  the  reasoning  process  of  the  ontological  problem-solving 
framework  to  determine  the  capabilities  of  sensors  and  algorithms  to  opportunistically  discover  and 
form  synthesized  systems  capable  of  satisfying  a  task. 

2.1 .  Ontological  Relationship  Structure 

hi  this  work,  the  descriptions  of  algorithms  and  sensors  are  represented  in  an  ontology  similar  to  the 
approach  taken  with  OntoSensor  and  CIEDETS,  which  were  developed  by  knowledge  engineers  with 
input  from  subject  sensor  matter  experts.  Using  OntoSensor  and  CIEDETS  ontologies  as  a  baseline  for 
the  ontological  reasoning  process,  the  ontology  needed  to  be  extended  to  allow  for  the  matching  of 
sensors  to  algorithms  to  form  synthesized  systems  capable  of  satisfying  a  task.  The  baseline  ontology 
was  extended  with  the  following:  (1)  a  class  hierarchy  for  describing  algorithms  with  descriptive 
properties;  (2)  additional  properties  in  the  sensor  class  for  describing  PFx  sensors;  (3)  an  additional 
class  hierarchy  for  matching  sensors  to  compatible  algorithms;  and  (4)  additional  declarative  rules. 

The  challenge  is  to  match  sensor  systems  to  algorithms  to  form  synthesized  systems  capable  of 
satisfying  a  task  and  then  reusing  those  systems  for  other  tasks.  The  baseline  ontologies  already 
describe  sensor  systems  and  various  properties  of  those  systems.  Since  the  focus  of  the  ontological 
problem-solving  framework  was  to  use  a  persistence  surveillance  sensing  environment,  properties 
were  added  to  the  sensor  classes  that  describe  PFx  sensor  systems.  Generally  these  systems  have 
properties,  such  as  image  resolution,  geo-locations  of  detectors  that  make  up  a  sparse  detector  array, 
and  network  communications  In  order  for  a  PFx  sensor  system  to  be  described  and  represented  by  the 
ontology,  these  properties  and  others  were  added  to  various  subclasses  of  the  Sensor  class.  Algorithms 
were  not  represented  by  the  baseline  ontologies  so  a  complete  class  hierarchy  was  added  along  with 
various  attributes,  such  as  data  input/output  requirements,  process  capabilities  and  purposes, 
descriptions  of  data,  and  network  communications  mapped  into  many  different  properties. 

If  sensor  systems  and  algorithms  are  matched  to  perform  a  task,  the  ontology  must  have  a  way  to 
describe  this  possible  interoperability.  This  combination  is  not  merely  just  a  sensor  and  a  compatible 
algorithm,  but  a  combination  of  systems  that  may  satisfy  a  given  task.  To  describe  this  possible 
combination  of  systems,  the  concept  of  a  synthesized  system  was  developed  and  integrated  into  the 
ontology.  A  synthesized  system  is  a  possible  combination  of  a  sensor  and  compatible  algorithm  that 
may  satisfy  a  task.  When  looking  at  various  types  of  sensor  and  algorithm  combinations  in  a 
persistence  surveillance  environment,  generally,  a  sensor  creates  raw  data  of  a  passing  object,  a  profile 
of  the  passing  object  is  created  from  the  raw  data,  and  then  the  profile  has  a  process  applied  to  it,  such 
as  a  classification  or  visualization.  This  is  a  two-step  process  of  first  generating  a  profile  and  second  to 
process  this  profile.  This  two-step  process  can  be  represented  by  two  different  synthesized  systems. 
The  first  synthesized  system  matches  a  sensor  to  an  algorithm  for  the  task  of  generating  profiles,  while 
the  second  synthesized  system  is  a  matching  of  the  first  synthesized  system  to  another  algorithm. 
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which  has  the  task  of  processing  the  profile  for  some  purpose.  To  represent  the  two  types  of 
synthesized  systems  in  the  ontology,  two  new  classes  were  created  that  have  object  type  properties  that 
establish  relationships  back  to  established  classes  and  properties.  Figure  10  shows  the  core  ontology  for 
matching  sensors  to  compatible  algorithms  to  form  synthesized  systems,  which  are  capable  of  satisfying 
a  task,  which  is  made  up  of  four  main  classes:  Matched _Sensor JSystem,  Profiling _Sensor_Sy stem. 
Sensor  and  Algorithm.  A  bottom-up  approach  wall  be  used  to  explain  the  purpose  of  each  of  the 
classes,  their  corresponding  relationships,  and  the  following  section  will  describe  the  rules  used  to 
queiy  the  ontology  instance  data  for  possible  synthesized  systems. 

Figure  10.  Core  ontology  of  the  ontological  problem-solving  framework  that  describes 
the  relations  of  the  classes:  Matched JSensorJSy stem,  Profiling_Sensor_System ,  Sensor 
and  Algorithm. 


The  Sensor  class  describes  a  sensing  device,  which  generates  raw  data.  The  Algorithm  class 
describes  a  process,  which  requires  raw  sensor  data  or  data  provided  by  another  algorithm  as  input  and 
then  generates  output.  The  Algorithm  class  can  include,  but  is  not  limited  to,  PFx  data  formatters,  PFx 
classifiers,  and  PFx  visualizers.  The  Profiling__Sensor_System  class  is  the  first  synthesized  system 
concept  that  describes  a  possible  combination  of  a  Sensor  instance  and  Algorithm  instance,  which 
produces  a  profile  of  an  object  in  the  sensor’s  field  of  view.  The  Sensor  and  Algorithm  instances  are 
linked  to  a  Profiling _Sensor_Sy stem  instance  through  the  two  object  type  properties  called  hasJSensor 
and  has_Algorithm.  A  Profiling_Sensor_System  may  have  many  Algorithm  instances  processing  the 
sensor  data.  For  example,  one  algorithm  may  extract  specific  pixels  from  a  raw  image  while  another 
algorithm  generates  a  profile  of  the  extracted  pixels,  thus,  a  chain  of  algorithms  and  sensors  may  be 
matched  in  a  Profiling_Sensor_System.  The  Matched JSensor ^System  class  is  the  second  synthesized 
system  concept  that  describes  a  possible  combination  of  a  Profiling_Sensor_System  instance  and 
Algorithm  instance,  which  produces  a  result,  such  as  a  visualization  or  classification  of  the  profile.  The 
instances  Profiling_Sensor_System  and  Algorithm  are  linked  to  a  Matched_SensorJ>ystem  instance 
through  the  object  type  property  has_Profiling_Sensor_System  and  has_  Algorithm.  A 
Matched _Semor_Sy stem  may  have  many  algorithms  processing  the  profile  from  the 
Profiling JSensor_fiystem  instance.  For  example,  one  algorithm  may  convert  the  profile  to  a  new 
format,  while  another  algorithm  operates  on  the  new  profile  to  generate  a  classification. 
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Figure  11.  Extended  class  hierarchy  of  the  ontological  problem-solving  framework  for  the 
Sensor  and  Algorithm  classes. 
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Figure  12.  Excerpt  of  the  properties  for  representative  classes  and  subclasses  for  the 
reasoning  process  in  the  ontological  problem-solving  framework. 


Figure  1 1  shows  the  class  hierarchy  of  the  Sensor  and  Algorithm  classes.  Each  of  these  classes  may 
have  many  properties,  which  are  used  to  describe  the  instances.  Figure  12  shows  several  of  the 
properties  used  to  describe  some  of  the  classes  within  the  ontology.  For  example,  the  subclass 
Photo -Conductive  of  the  Sensor  class  has  specific  properties  describing  a  sensor's  pixel  resolution: 
has_Horizontal_Pixel_Resolution  and  has_Vertical_Pixel_Resolution  while  also  inheriting  the  Sensor 
class  property  has_Network_Communication.  The  subclass  Pixel-Extractor  of  class  Algorithm  has 
properties  describing  the  resolution  of  a  generated  profile:  has_Input_Horizontal_Resolution  and 
has_Input_Vertical_Resolution  while  also  inheriting  the  property  has_Network_Communication  from 
the  Algorithm  class.  Similar  in  nature  is  the  subclass  Naive_Bayes_Classifier  which  inherits  from  the 
same  Algorithm  class  but  also  adds  its  own  unique  properties  such  as  has_Classification_Target.  The 
Profiling_Sensor_System  and  Matched_Sensor System  classes  also  have  properties,  which  are  derived 
from  the  Sensor  and  Algorithm  classes  through  rules  executed  during  the  inference  process.  These 
object  and  data  type  properties  are  only  a  few  of  tire  many  describe  in  the  ontology. 
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2.2.  Ontological  Rules 

The  graph-matching  query  language  SPARQL  [8]  was  used  to  create  declarative  rules  for  the 
ontological  problem-solving  framework.  The  SPARQL  query  language  has  internal  functions  that  will 
allow  for  the  querying  of  possible  synthesized  systems  through  an  inference  engine.  Once  the 
synthesized  systems  are  returned  back  from  the  inference  engine  the  systems  can  be  fonned  into  new 
instance  data  to  be  leveraged  by  other  systems  on  the  ontological  problem-solving  framework.  The 
rules  contain  statements  that  consist  of  logical  constraints  among  instance  data  and  properties  that  must 
be  true  for  subsequent  instances  and  properties  to  be  derived  and  returned  as  results  back  to  the 
ontology.  The  rules  are  made  up  of  two  components,  referred  to  as  the  WHERE  and  CONSTRUCT 
clauses.  The  CONSTRUCT  (Figure  13(a))  clause  is  used  to  return  possible  object  instances  and 
properties  based  on  instance  data  and  properties  that  satisfy  the  WHERE  clause  of  the  SPARQL  rule. 
The  returned  instances  may  include  links  to  established  instances  (Figure  13(b,c)),  as  well  as  links  to 
derived  attributes  of  the  returned  instances.  The  WHERE  clause  contains  the  logical  constraint 
statements  that  queried  existing  instances  must  satisfy  before  the  CONSTRUCT  clause  returns  the 
possible  instances  and  establishes  links  to  the  pre-existing  instances  and  properties  (Figure  14).  The 
WHERE  clause  constraint  statements  include  preconditions  (properties  that  must  exist),  and  the  other 
descriptive  logical  constraints,  such  as  FILTER  and  OPTIONAL  statements,  that  existing  queried 
instances  must  satisfy*  before  possible  instances  and  properties  are  returned  by  the  CONSTRUCT 
clause.  Each  rule  can  be  regarded  as  a  Horn  clause  in  that  each  condition  is  specified  in  the  rule  via 
logical  conjunction  (logical  AND).  If  all  the  properties  hold  true  then  the  specified  instance  is  returned 
by  the  rule.  Logical  disjunction  (i.e.,  logical  OR)  can  be  regarded  as  a  collection  of  rules  that  create  a 
similar  instance,  for  example,  a  collection  of  rules  that  each  bind  on  different  properties  which  return 
instances  of  a  Profiling_Sensor_System . 

Figure  13.  SPARQL  CONSTRUCT  clause  (a)  Returned  MatchedJSensorJSystem 
instance,  histance_Matched_Sensor_System,  linked  to  Sensor  and  Algorithm  instances. 

(b)  Instance_Sensor  and  (c)  Instance_Algorithm  variables  instantiated  to  specific  Sensor 
and  Algorithm  instances  in  the  WHERE  clause,  thereby  establishing  a  link  between  a 
matched  Sensor  instance  and  an  Algorithm  instance,  (d)  Instance  diagram. 
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Figure  14.  SPARQL  WHERE  clause  (a)  The  variable  Instance_Sensor  instantiated  to  an 
instance  of  tire  class  Sensor  with  the  data  property  (b)  has_Type  established  to  the  variable 
Type_Sensor.  (c)  The  variable  Instance_Algorithm  instantiated  to  an  instance  of  the  class 
Algorithm  with  tire  data  property  (d)  has_Type  established  to  the  variable  Type_Algontlrnr. 
(e)  FILTER  command  comparing  Type_Sensor  and  Type_Algoritlrm  variables  for 
compatibility,  (f)  Instance  diagram. 
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The  inference  engine  will  process  the  SPARQL  rules  for  all  combinations  of  pre-existing  instances. 
For  example,  in  Figure  14(a,c),  these  two  statements  result  in  the  WHERE  clause  cycling  through  all 
Sensor  and  Algorithm  instances.  The  statements  in  Figure  14(b,d)  bind  the  property  has_Type  value  for 
the  instances.  The  FILTER  statement  in  Figure  14(e)  compares  the  value  of  has_Type  for  the  Sensor 
and  Algorithm  instances.  If  the  FILTER  statement  is  satisfied,  then,  the  CONSTRUCT  clause  is 
subsequently  executed  to  return  the  specified  instance  and  associated  properties.  For  a  simple  example, 
the  instance  data  in  Figure  15  will  be  queried  with  a  complete  SPARQL  rule  with  the  CONSTRUCT 
and  WHERE  clauses  in  Figures  13  and  14.  The  Photo ^Conductive  sensor  instance  and  Pixel_Extractor 
algorithm  instance  each  have  the  property  has_Type  with  a  value  of  “Image'’  (Figure  15(a)).  When  the 
complete  SPARQL  rule  of  Figures  13  and  14  is  executed  by  tine  inference  engine  the  WHERE  clause 
will  query  for  a  possible  Sensor  and  Algorithm  instances  whose  property  has_Type  are  the  same 
(Figure  15(b)).  Once  a  possible  combination  has  been  found  {Photo ^Conductive  and  Pixel  _Extr  actor 
in  this  case),  the  CONSTRUCT  clause  will  be  execute  by  the  inference  engine  to  return  the  possible 
Matched JSensor JZy  stem  instance  with  links  back  to  the  original  Photo JC onductive  and 
Pixel JExtr actor  instances  (Figure  15(c)).  The  returned  Matched JSensorJSy stem  instance  will  then  be 
placed  into  the  ontology  for  further  inference  and  use  by  other  systems.  Even  though  this  is  a  simple 
example  with  SPARQL,  with  additional  constructs,  such  as  the  FILTER  or  OPTIONAL  commands, 
far  more  complex  rules  may  be  built. 
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Figure  15.  Instance  diagram  of  a  SPARQL  query  binding  on  specific  instance  data  and 
returning  possible  instances  (a)  Existing  Sensor  and  Algorithm  instances  that  have 
has_Type  values  equal  to  “Image”  (b)  WHERE  clause  binding  and  checking  tire  has_Type 
property  (c)  CONSTRUCT  clause  returning  a  possible  Matched_Sensor_System  with 
established  links  to  the  found  Sensor  and  Algorithm  instances. 


Sensor 


(a)  (b)  (c) 

Tire  rules  in  tire  ontological  problem-solving  framework  bind  on  all  combinations  of  Sensor  and 
Algorithm  instances.  Their  respective  properties  are  then  compared  in  the  FILTER  statements  of  the 
WHERE  clause  to  detennine  which  instances  need  to  be  returned  and  when  to  establish  links  between 
other  instances.  Figure  16  through  Figure  19  each  show  one  of  many  rules  used  to  return  possible 
Profiling_Sensor_System  instances  and  Matched_Sensor_System  instances.  The  WHERE  clause  in  the 
Profiling_Sensor_System  rules  in  Figures  16  and  17  bind  on  the  properties  of  Sensor  and  Algorithm 
instances,  such  as  pixel  resolution  in  Figure  16,  number  of  detectors  in  Figure  17,  and  type  for  both 
Figures  16  and  17.  Further,  in  the  WHERE  clause,  the  FILTER  statement  now  compares  specific 
Sensor  instance  properties  to  the  Algorithm  instance  properties.  For  example,  in  Figure  16,  the  FILTER 
statement  compares  the  network  communication  type  and  pixel  resolutions.  Once  a  set  of  instances  for 
a  Sensor  and  Algorithm  have  been  queried,  which  satisfy  the  constraints  of  the  WHERE  clause,  the 
CONSTRUCT  clause  will  then  return  a  Profiling_Sensor_System  instance  and  establish  links  to  the 
compatible  Sensor  and  Algorithm  instances.  The  same  process  occurs  in  the  WHERE  clause  in 
Figure  17,  but  instead  of  comparing  pixel  resolutions,  detector  properties  are  compared  for 
compatibility.  The  rules  for  Matched_Sensor_System  in  Figures  18  and  19  follow  a  similar  logical 
process  as  the  Profiling_Sensor_System  rule.  The  only  difference  between  the  rules,  other  than  the 
specific  properties  of  the  instances,  is  in  the  FILTER  statement  where  an  additional  statement 
constrains  the  WHERE  clause  to  a  specific  type  of  Algorithm ,  in  this  case  a  “Classifier”.  The  rules 
shown  in  Figures  14  and  19  both  return  Matched _Sensor _Sy stem  instances,  which  will  classify  the 
generated  profiles  of  Profiling_Sensor_System  instances. 
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Figure  16.  Sample  rule  and  instance  diagram.  The  rule  returns  an  instance  of  a 
Profiling_Sensor_System  if  the  Algorithm  instance  and  Sensor  instance  are  type  compatible 
with  respect  to  the  network  communication  and  pixel  resolutions  properties. 
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Figure  17.  Sample  Profiling_Sensor_Syslem  rule  and  instance  diagram.  The  rule  returns  an 
instance  if  the  Algorithm  instance  and  Sensor  instance  properties:  type,  network 
communication,  number  of  detectors,  and  displacement  properties  are  compatible. 
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Figure  18.  Sample  Malched_Sensor_System  rule  and  instance  diagram.  The  rule  returns  an 
instance  if  tine  Profiling_Sensor_Syslem  instance  and  Algorithm  instance  properties:  network 
communication,  types,  encoding,  classification,  and  pixel  resolutions  are  compatible. 
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Figure  19.  Sample  Matched_Sensor_System  rule  and  instance  diagram,  which  returns  an 
instance  if  tire  ProJiling_Sensor_System  instance  and  Algorithm  instance  properties: 
network  communication,  types,  encoding,  classification,  data  rows,  and  columns  properties 
are  compatible. 
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'5lnstance_Profiling_Algortthm 
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?lnstance_Profiling_Algorithm 
?lnstance_Profiling_Algorithm 
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?lnstance_Algorithm 
?lnstance_Algorithm 
?lnstance_Algorithm 
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?lnstance_Algorithm 
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23.  Instances  of  Profiling  Sensor  Systems  and  Algorithms  on  Ontological  Problem-Solving  Framework 

To  illustrate  a  simple  case,  Figure  20  shows  five  sensor  instances,  including  three  PFx  sensors  and 
two  conventional  imagers,  and  six  algorithms,  including  two  profile  generators  and  four  different 
classifiers,  with  different  property  specifications  and  requirements.  When  the  inference  cycle  begins, 
the  rules  from  Figure  16  through  Figure  19  will  execute.  On  the  first  pass  of  the  inference  cycle,  five 
new  Profiling _Sensor_System  instances  were  created,  as  shown  in  Figure  21.  The  two  algorithms 
Profile  Image  Generator  and  Profile  Matrix  Data  Generator  were  matched  to  multiple  sensors  based  on 
constraints  of  the  algorithms  and  specifications  of  the  sensors.  For  example,  the  Algorithm  instance 
Profile  Image  Generator  was  matched  to  the  Sensor  instance  PF5  Conventional  Visible  Imager  because 
the  constraint  of  requiring  image  data  for  the  Profile  Image  Generator  was  satisfied. 

Figure  20.  Example  instances:  (a)  Three  PFx  sensors  and  two  conventional  imaging 
sensors,  (b)  Two  profile  generators  and  four  classifiers. 


A)  Sensor  Instance  Data 


B)  Algorithm  Instance  Data 


Sensor:  PFt  Sparse  Detector  with  16  NIR  detector  spaced  12  inches 


Sensor:  PF?  Sparse  Detector  with  16  NIR  detector  spaced  20  inches 


Sensor:  PF,  Sparse  Detector  with  8  Thermopile  detectors 


Sensor:  PF4  Conventional  Visible  Imager  with  640  x  460  resolution 


Sensor:  PF}  Conventional  MWIR  Imager  with  640  x  480  resolution 


Algorithm:  Profile  Image  Generator 
Input  Image 
Output  Profile  Image 


Algorithm :  Profile  Matrix  Data  Generator 
Input  Vertcal  Colum  n  Data 
Output  Profile  T ext  Data 


Algorithm:  Classifier  Human 
Input :  Profile  T ext  Data 


Algorithm:  Classifier  Human 
Input;  Profile  Image 


Algorithm:  Classifier  Vehicle 
Input;  Profile  Image  or  Text  Data 


Algorithm:  Classifier  Animal 
Input :  P  rof ile  T  ext  Data 


Figure  21.  Five  new  Profiling JlensorJSystem  instances  returned,  with  derived  relationships, 
after  the  first  pass  of  the  inference  cycle. 
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Figure  21.  Coni. 


ProfileSensofiSyelem:^ 


Sensor:  PF„  Sparse  Detector  with  8  Thermopile  detectors 


Algorithm:  Profile  Matrix  Data  Generator 
Input  :  Vertical  Column  Data 
Output.  Profile  Text  Data 


•-^ProfileiSensoriSystem:  -PF*  matched  Profile  I  mage  <3enerator. 


Se nsor:  PF,  Conventional  MWIR  Imager  with  640  x  480  resolution 


Algorithm:  Profile  Image  Generator 


Input  : 
Output: 


Image 

Profile  Image 


Profile_Sensor_System:  PF*  matched  Profile  ImageGenerator 


f  Sensor:  PF„  Conventional  Visible  Imager  with  640x480  resolution 


Algorithm:  Profile  Image  Generator 
Input  :  Image 
Output:  Profile  Image 


During  the  second  pass  of  the  inference  cycle,  thirteen  new  Matched JSensorJSystem  instances  were 
created,  as  shown  in  Figure  22.  The  four  different  classifiers  were  matched  to  multiple 
Pr  o filing  _Sensor__Sy stem  instances  based  on  the  type  of  profile  generated  and  the  requirements  of  the 
classifiers.  For  example,  the  Profiling  JSensorJSy stem  instance  PFj  matched  Profile  Data  Generator 
was  matched  to  the  Algorithm  instance  Human  Classifier  because  the  constraint  of  requiring  text  data 
was  satisfied  for  the  Human  Classifier.  On  the  third  pass  of  the  inference  cycle,  no  new  instances  were 
created;  therefore,  the  inference  cycle  halts  and  returns  all  matches. 

Figure  22.  Thirteen  new  Matched_Sensor_System  instances  returned,  with  derived 
relationships,  after  the  second  pass  of  the  inference  cycle. 
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Figure  22.  Coni. 


4V!atchecLSen8or_System  tPF^Profile  Data  Generator^tchedClassifierwutnan 


S  «tQfilft5Serwr  system 


tfi  mxm&m  Gonefatcr 


Algorithm;  Classifier  Human 
Input  :  Profile  Text  Data 


Sensor:  PF,  Sparse  Detector  with  16  NIR  detector  spaced  20  inches  | 


Algorithm;  Profile  Matrix  Data  Generator 
Input  :  Vertical  Column  Data 
Output:  Profile  Text  Data 


Match  etLSehsofJSystenh  PK,  Profile  Bate  Generator  ^matched  Classified  Vehicle 


E 


mWk  Sensotsystemf  PF,  r^tchetfPtofileBateiCeherator 


Algorithm:  Classifier  Vehicle 
Input  ;  Profile  Image  or  Text  Data 


Sensor:  PF,  Sparse  Detector  with  16  NIR  detector  spaced  20  inches  1 


Algorithm:  Profile  Matrix  Data  Generator 
Input  :  Vertical  Column  Data 
Output:  Profile  Text  Data 


Match  ed5ensor,System:PK,P/oFile  DateGeneratorrtiatchedClassffierAnimal 


id  ProfileBate  Generator 


Algorithm:  Classifier  Animal 
Input  :  Profile  Text  Data 


Sensor:  PF7  Sparse  Detector  with  16  NIR  detector  spaced  20  inches 


Algorithm:  Profile  Matrix  Data  Generator 
Input  :  Vertical  Column  Data 
Output:  Profile  Text  Data 
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Figure  22.  Cont. 


Matched  jSensocSystem:  PF<  Profile  Image  Generator  jnatched  Classifier  frluman 


Algorithm:  Classifier  Human 
Input  Profile  Image 


Sensor:  PF^  Conventional  Visible  Imager  with  640  x  480  resolution 


Algorithm:  Profile  Image  Generator 
Input  :  Image 
Output:  Profile  Image 


3.  Discussion 

The  challenge  was  to  match  sensor  systems  to  compatible  algorithms  to  fonn  synthesized  systems, 
which  are  capable  of  satisfying  a  task  and  matching  those  systems  to  new  systems  for  other  tasks.  The 
sample  rules  described  in  this  paper  specified  relatively  simple  compatibility  constraints  among 
sensors  and  algorithms.  However,  even  with  these  simple  rules,  it  is  noteworthy  that  the  Algorithm 
instances  were  matched  to  multiple  Sensor  and  Pro fi ling_Sensor_System  instances  thus  achieving  the 
ability  to  reuse  those  systems  for  tasks  that  may  have  not  been  anticipated  at  the  time  the  sensors  and 
algorithms  were  first  deployed.  For  example,  of  the  five  synthesized  system  concept 
Profiling_SensorJSyslem  instances  that  were  returned,  the  algorithm  Profile  Matrix  Generator  was 
matched  to  three  different  sensor  systems  and  the  algorithm  Profile  Image  Generator  was  matched  to 
two  sensor  systems.  If  not  for  the  matching  and  return  of  the  Profiling JSensorJSy stem  synthesized 
systems,  each  one  of  the  matched  systems  would  have  had  to  be  individually  designed. 

The  same  results  can  be  seen  in  the  synthesized  system  Matched _S enso rJSystem ,  which  reused  the 
five  Pro filing_Sensor_Sy stem  synthesized  systems  in  thirteen  systems  with  different  tasks,  such  as 
visualizing  or  classifying  the  profiles.  If  the  original  algorithms  represented  by  the  Algorithm  instances 
had  been  designed  for  specific  Sensor  instances,  the  reasoning  process  of  the  ontological 
problem-solving  framework  would  not  have  matched  the  algorithms  to  new  sensors,  thus  the  sensor 
systems  and  algorithms  would  have  had  to  be  re-engineered  specifically  for  one  another  to  satisfy  a 
task.  It  is  important  to  note  that  the  synthesized  system  concepts  Pro filing_Sensor_Sy stem  and 
Matched_Sensor_System  capture  more  than  just  a  Sensor  matched  to  an  Algorithm.  The  concept 
synthesized  systems,  represent  new  systems  which  are  capable  of  performing  a  task.  Other  rules  in  the 
ontological  problem-solving  framework  may  operate  on  far  more  than  just  two  attributes  for 
establishing  interoperability  via  matching  constraints.  The  rules  may  determine  that  multiple  matched 
Profi  li  ng  JSensorJSy  stem  and  Matched _Sensor_Sy  stem  instances  may  be  fonned  into  new  more 
complex  synthesized  systems,  which  may  be  capable  of  satisfying  more  complex  tasks,  which  may 
include  statistical  analysis  on  multiple  profiles.  With  the  formation  of  the  synthesized  system  by  the 
reasoning  process,  the  ontological  problem-solving  framework  may  create  more  complex  synthesized 
systems.  These  more  complex  systems  may  then  be  assigned  to  subtasks  of  high-level  missions  by 
other  systems  on  the  network  coordinating  and  executing  the  mission.  Without  the  use  of  the  ontology, 
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rules,  and  inference  engine  these  sensors  and  algorithms  would  have  had  to  be  designed  a  priori  a 5  a 
synthesized  system  for  every  new  task.  However,  many  of  these  new  tasks  are  not  known  at  the  time 
the  systems  are  deployed;  therefore,  opportunistically  discovering  compatible  systems  and  dynamically 
creating  matched  synthesized  systems  which  are  capable  of  satisfying  a  new  task  through  inference 
is  important. 

Currently,  the  reasoning  process  of  the  ontological  problem-solving  framework  is  still  in  a 
prototype  stage  so  scale-up  performance  analysis  is  limited.  The  problem-solving  framework  can  scale 
to  large  numbers  of  sensors  and  algorithms,  but  the  time  to  compute  all  combinations  of  sensors  and 
algorithms  is  based  on  the  computational  complexity  of  the  inference  engine,  which  is  influenced  in 
part  by  the  reasoning  strategy  and  the  expressiveness  of  the  knowledge  representation  formalism.  For 
the  perfonnance  to  increase,  the  inference  engine  must  check  multiple  algorithms  in  parallel  or  the 
ontological  problem-solving  framework  must  invoke  the  inference  engine  multiple  times  in  parallel 
with  different  algorithms  and  keep  track  of  which  instances  are  being  checked  to  stop  redundant 
bindings.  Even  though  the  ontological  problem-solving  framework  is  still  in  the  prototype  stage, 
perfonnance  issues  and  solutions  are  being  studied;  however,  the  logical  framework  is  the  priority  at 
this  stage  of  research. 

4.  Conclusions 

Challenges,  such  as  matching  sensors  to  compatible  algorithms  that  may  satisfy  a  task,  will  become 
even  more  difficult  with  the  continued  development  and  deployment  of  new  sensor  systems  and 
algorithms.  Compounding  the  challenge  is  the  lack  of  knowledge  models  used  to  explicitly  capture  the 
design  and  capabilities  of  sensor  systems  and  algorithms.  By  leveraging  knowledge  models,  sensor 
systems  and  algorithms  can  be  matched  together  in  real-time  without  the  need  to  design  those  matched 
systems  specifically  for  one  another  a  priori ,  thus  facilitating  the  use  of  these  assets  in  new  and 
innovative  ways  not  originally  anticipated  on  deployment.  To  exploit  the  power  of  knowledge  models, 
algorithms  must  become  less  dependent  on  any  given  sensor  data  source,  thus  sensor  systems  and 
algorithms  must  describe  their  respective  attributes  and  capabilities  in  a  machine-interpretable  format 
to  allow  the  reasoning  process  to  infer  which  systems  may  be  matched  together  into  more  complex 
synthesized  systems.  The  reasoning  process  of  the  ontological  problem-solving  framework  discussed 
in  this  paper  is  the  first  step  to  achieving  this  goal  and  addressing  the  challenge  of  matching  systems 
that  are  capable  of  satisfying  a  task.  Even  though  the  reasoning  process  of  the  ontological 
problem-solving  framework  was  described  in  the  context  of  profiling  sensor  systems  and  algorithms, 
the  overall  approach  may  be  used  for  other  types  of  sensor  systems  and  algorithms  to  form  different 
types  of  synthesized  systems  capable  of  satisfying  new  tasks. 
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Abstract:  The  lack  of  knowledge  models  to  represent  sensor  systems,  algorithms,  and 
missions  makes  opportunistically  discovering  a  synthesis  of  systems  and  algorithms  that 
can  satisfy  high-level  mission  specifications  impractical.  A  novel  ontological  problem-solving 
framework  has  been  designed  that  leverages  knowledge  models  describing  sensors, 
algorithms,  and  high-level  missions  to  facilitate  automated  inference  of  assigning  systems 
to  subtasks  that  may  satisfy  a  given  mission  specification.  To  demonstrate  the  efficacy  of  the 
ontological  problem-solving  architecture,  a  family  of  persistence  surveillance  sensor 
systems  and  algorithms  has  been  instantiated  in  a  prototype  environment  to  demonstrate  the 
assignment  of  systems  to  subtasks  of  high-level  missions. 

Keywords:  sensor  networks;  Sensor  Ontology;  profiling  sensors;  mission  tasking 


1.  Introduction 

Dynamically  discovering,  matching,  and  integrating  sensors  and  compatible  algorithms  to  form  a 
synthesis  of  systems  that  are  capable  of  satisfying  subtasks  of  high-level  missions  poses  a  significant 
challenge  for  network-centric  architectures.  Compounding  the  challenge  is  the  lack  of  knowledge  and 
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data  models  used  to  describe  the  relationships  among  sensors,  algorithms,  and  missions.  Most 
algorithms  are  designed  for  specific  sensor  systems  in  anticipation  of  performing  a  specific  task. 
Designing  and  deploying  tightly  integrated  systems  limits  their  potential  reuse  for  new,  unanticipated 
tasks  without  re-engineering  the  systems  [1-6].  This  paper  will  review  the  authors’  pnor  work  [7], 
which  addresses  the  issue  of  autonomously  matching  sensor  systems  to  compatible  algorithms.  Section  2 
of  the  paper  will  review  the  challenges  of  assigning  the  matched  systems  to  subtasks  of  missions. 
Section  3  will  review  related  work  of  systems  and  frameworks  that  assign  systems  to  missions.  The 
remainder  of  the  paper  will  focus  on  the  authors’  extension  of  their  previous  work  to  now  include 
assignment  of  the  synthesis  of  systems  to  subtasks  of  missions  in  the  context  of  a  persistence 
surveillance  sensing  environment.  Section  4  discusses  the  operation  of  the  persistence  surveillance 
environment  and  Sections  5,  6,  and  7  discuss  the  extended  ontological  problem-solving  framework 
laboratory  prototype  for  mission  assignment  and  execution. 

Previous  Work  by  Qualls  and  Russomanno 

Matching  sensor  systems  to  compatible  algorithms  to  form  a  synthesis  of  systems  poses  a 
significant  challenge  to  problem-solving  frameworks.  Frameworks  must  be  able  to  match  the  systems 
together  and  then  reuse  the  same  systems  in  new  matches  as  depicted  in  Figure  1.  In  prior  work,  Qualls 
and  Russomanno  [7]  focused  on  the  reasoning  process  of  matching  sensor  systems  and  algorithms  to 
form  a  synthesis  of  systems  capable  of  satisfying  a  task. 

Figure  1.  Process  for  matching  sensor  systems  to  compatible  algorithms  to  form  a  synthesis 

of  systems  capable  of  satisfying  a  task. 


The  prior  work  by  the  authors  included  developing  a  laboratory  prototype  ontological 
problem-solving  framework  that  leveraged  knowledge  engineering  techniques  to  opportunistically 
infer  the  discovery  and  matching  of  sensor  systems  to  compatible  algorithms.  The  knowledge 
engineering  techniques  included  an  ontology,  rules,  and  inference  engine  to  autonomously  fonn  the 
synthesis  of  systems.  Standard  database  technologies  and  SQL  queries  could  have  been  used  for  the 
prototype  development,  but  one  of  the  main  shortcomings  limiting  the  matching  of  systems  together  is 
the  lack  of  knowledge  models  to  describe  and  represent  the  systems.  The  knowledge  models 
themselves  must  leverage  well-defined  semantics  in  a  machine-interpretable  format  for  autonomous 
matching.  The  use  of  knowledge  models  also  provides  the  added  benefit  of  more  readily  transferring 
the  knowledge  to  other  systems  as  compared  to  other  techniques. 

To  autonomously  fonn  the  synthesis  of  systems,  the  prototype  framework  used  ontologies  to 
descnbe  properties  and  relationships  among  sensor  systems,  algorithms,  and  possible  synthesis  of 
systems.  The  ontologies  have  two  parts:  (i)  the  class  hierarchy  for  describing  relations  among  different 
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types  of  sensor  systems;  and  (ii)  algorithms  and  properties  for  describing  specific  properties  of  each 
class.  Data-type  properties,  which  may  be  regarded  as  attributes,  are  used  to  describe  sensor  system 
and  algorithm  parameters,  such  as  pixel  resolutions,  field  of  view,  data  format,  algorithmic  process, 
and  network  connections.  Object-type  properties,  which  may  be  regarded  as  associations,  were  used  to 
link  specific  sensor  systems  and  algorithms  together  during  the  inference  process  for  the  synthesis  of 
systems.  With  the  properties  in  the  ontology,  instance  data  may  then  be  created  to  represent  actual 
sensor  systems  and  algorithms. 

Figure  2  shows  a  small  excerpt  of  the  ontology  including  four  main  classes  for  synthesis: 
Matched_Sensor J$y>stem,  Profiling_Sensor_System ,  Sensor ,  and  Algorithm.  The  Sensor  class  describes 
a  sensing  device  and  the  Algorithm  class  describes  an  algorithmic  process.  The  process  can  either 
operate  on  data  generated  by  the  sensing  device  or  data  generated  by  other  processes.  The 
Profiling_Sensor_System  class  represents  a  synthesis  of  systems  that  describes  possible  combinations 
of  Sensor  and  Algorithm  instances  which  produce  formatted  signal  profiles  of  objects  as  they  pass 
through  a  sensor  system's  field  of  view.  The  class  Matched _Sensor_System  describes  a  synthesized 
system  that  contains  possible  combinations  of  Profiling jSensor^System,  Algorithm ,  and  Sensor 
instances,  which  produce  results,  such  as  visualizations  or  classifications  of  the  generated  signal 
profiles.  Not  shown  is  the  class  hierarchy  for  the  Target  class,  which  contains  further  subclasses  of 
Human ,  Animal  and  Vehicle.  These  subclasses  are  further  refined  and  include  subclasses,  such  as  of 
Bird,  Large _Animal,  and  Bear  for  Animal  and  subclasses  of  Car ,  Light JTruck,  and  Heavy _Truckt  for 
Vehicle ,  and  so  on.  Also  not  shown  is  a  class  hierarchy  of  the  Object _Oj ^Interest,  which  includes 
subclasses  and  properties  describing  accessories,  such  as  backpacks  and  an  extensive  description  of 
weapons,  which  include  bladed,  non-bladed  weapons,  and  projectile  weapons,  such  as  small  and  heavy 
arms  including  pistols,  machine  guns,  and  rocket-propelled  grenades.  Each  of  the  further  subclasses 
has  its  own  respective  data-type  properties  describing  those  classes.  Rules  in  the  form  of  queries  with 
conditional  actions  were  developed  to  be  processed  by  an  inference  engine  to  search  the  instance  data 
for  possible  synthesis  of  systems.  For  further  information  on  the  development  of  the  ontology  in 
Figure  2,  class  hierarchy,  and  rule  design  please  refer  to  Qualls  and  Russomanno  [7]. 

Figure  2.  Excerpt  of  the  ontology  in  the  ontological  problem-solving  framework  for 

matching  sensors  to  algorithms  to  form  a  synthesis  of  systems. 


has_Profiling_Sensor_System  1 


has_A1gorlthm  1  ..  * 


has_Sensor  1  .. 


The  ontological  problem-solving  framework  with  the  knowledge  engineering  techniques  discussed 
above  was  developed  w'ith  the  TopBraid  Composer  Maestro  software  environment  by  TopQuadrant  [8], 
TopBraid  uses  the  Web  Ontology  Language  (OWL)  [9]  for  authoring  ontologies;  rules  with  logical 
conditions  were  developed  with  SPARQL  [10];  and  the  TopSPIN  inference  engine  was  used  for 
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processing  the  SPARQL  rules.  The  authors  chose  TopBraid  Composer  Maestro  due  to  their  familiarity 
with  this  platform  from  other  research  projects.  OWL  is  based  upon  one  specific  description  logic 
(DL)  with  the  main  difference  being  the  naming  nomenclature.  For  example,  in  OWL  a  class  is  a 
concept  in  DL,  an  OWL  property  is  a  role  in  DL,  and  an  OWL  object  is  an  individual  in  DL.  The  rules 
developed  by  the  authors  in  the  ontological  problem-solving  framework  are  expressed  as  SPARQL 
queries  with  additional  constraints  on  RDF  triples,  hi  addition,  some  of  the  rules  include  actions 
implemented  by  invoking  Java  functions  via  procedural  attachment  [11],  Other  ontological  development 
environments  could  have  been  used  for  the  prototype  development,  such  as  Protege  with  JESS  and 
SWRL  [1-7].  Figure  3  shows  the  overall  framework  of  the  ontological  problem-solving  system. 


Figure  3.  Overview  of  the  laboratory  prototype  ontological  problem-solving  framework. 


Ontological  Problem-Solving  Framework 


2.  Assigning  Systems  to  Missions 

TTie  prior  prototype  ontological  problem-solving  framework  developed  by  the  authors  only  matched 
sensors  and  algorithms  to  form  a  synthesis  of  systems  [7].  The  next  logical  step  was  to  extend  the 
framework  to  allow  for  missions  to  be  instantiated  on  the  framework  and  then  autonomously  assign  the 
synthesis  of  systems  to  the  missions.  Before  an  extension  could  be  made,  the  concept  of  a  mission 
must  be  developed.  Knowledge  acquired  from  subject  matter  experts  (SMEs)  in  the  fields  of  sensor 
system  design,  algorithm  development,  and  concept  of  operations  (CONOPS)  contributed  to  the 
development  of  the  concept  of  missions.  The  authors  elicited  knowledge  from  the  SMEs  to  first 
develop  missions  associated  with  typical  persistence  surveillance  applications  as  illustrated  in  Figure  4. 

Figure  4.  Typical  missions  for  persistence  surveillance. 


^classify  human  carrying. sub-machine  gun 


I  visualize  human  carrying  a  ! 
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The  authors  and  SMEs  then  analyzed  the  typical  missions  to  yield  a  set  of  specifications  that  could 
be  used  to  describe  the  missions.  The  specifications  included  a  target  that  must  be  detected,  such  as  a 
human  or  vehicle,  and  a  mission  task  for  describing  a  subtask  that  describes  a  process  and  condition 
which  takes  place  on  the  target,  A  process  describes  what  must  happen  on  the  detected  target,  such  as 
“classification”,  “visualization”,  or  “signal  profile  generation”.  The  process  may  have  ancillary 
conditions,  such  as  target  is  carrying  a  “weapon”  or  “backpack”  or  even  a  condition  of  target  has 
“weight  greater  than  six  tons”.  A  specification  is  shown  in  Figure  5  via  an  instance  diagram. 

Figure  5.  Mission  decomposed  into  a  specification  via  an  instance  diagram. 


Figure  6.  (a)  Decompose  missions  to  separate  subtasks;  (b)  Discover  sensors  and 
compatible  algorithms,  which  can  complete  subtasks;  (c)  Subtask  assigned  to  a  chain  of 
algorithms  operating  on  raw  data  produced  by  a  sensor;  (d)  Subtask  assigned  to  an 
algorithm  operating  on  raw  sensor  data  from  two  different  sensors,  and  (e)  Subtask 
assigned  to  an  algorithm  operating  on  raw  data  produced  by  a  sensor. 


With  the  high-level  missions  decomposed  into  a  set  of  mission  specifications,  a  problem-solving 
approach  must  then  assign  matched  sensor  systems  and  algorithms  to  the  subtasks  which  satisfy  the 
mission  specification  as  indicated  in  Figure  6.  The  problem-solving  approach  must  discover  which 
systems  can  satisfy  the  given  subtasks  as  illustrated  in  Figure  6(b).  Once  systems  have  been 
discovered,  the  interoperation  of  multiple  sensors  and  algorithms  must  be  coordinated  to  perform  a 
subtask  as  indicated  in  Figure  6(c-e).  To  perfonn  these  operations,  the  ontological  framework  must 
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describe  relationships  among  the  sensors,  algorithms,  and  missions  in  terms  of  how  the  subtasks  relate 
to  the  mission,  relationships  among  subtasks  and  compatible  sensors  and  algorithms,  and  the 
relationships  between  the  sensors  and  algorithms.  Section  5  details  how  the  developed  concept  of  a 
mission  was  integrated  into  the  authors’  previous  work. 

3.  Related  Work 

Various  approaches  have  been  designed  and  engineered  to  assign  sensors  and  algorithms  to 
missions  or  task  specifications,  such  as  Semantic  sensor  mission  assignment  [12],  Ontological  sensor 
mission  assignment  [13],  Knowledge  base  for  sensors  to  missions  [14],  GloServ  [15],  Ontology 
Centric  sensor  mission  assignment  [16],  Resource  management  [17],  Sensor  Mission  Matching  [18], 
Semantic-aware  cooperative  agents  [19],  Query  Processing  for  sensor  networks  [20],  Agilla  [21-23], 
Geographic  Information  System  Framework  [24,25],  Semantic  Streams  [26,27],  and  Sensor  OASiS  [28], 
Relevant  to  our  work  is  the  development  of  ontologies  that  represent  and  describe  sensor  systems  such 
OntoSensor  [2-6],  Sensor  Network  Data  Ontology  [29],  Sensor  and  Data  Wrapping  Ontology  [30], 
Stimulus-Sensor-Observation  Ontology  [31],  Sensor  Observation  and  Measurement  Ontology  [32], 
Semantic  Sensor  Network  Ontology  [33],  and  Disaster  Management  Sensor  Ontology  [34].  Also  of 
importance  to  the  authors’  work  are  other  examples  of  relevant  sensor  ontologies  [35].  There  are  many 
logical  models  and  standards  to  follow  and  adapt,  such  as  the  Sensor  Modeling  Language 
(SensorML)  [36],  that  leverage  the  Unified  Modeling  Language  (UML)  to  conceptualize  sensor 
systems  and  algorithms  to  facilitate  interoperability.  Also,  the  Open  Geospatial  Consortium  (OGC)  [37] 
drafts  standards  that  may  be  used  to  define  metadata  encodings  and  interoperability  interface  standards 
to  facilitate  problem-solving  frameworks  that  can  integrate  sensor  systems  and  algorithms  into 
information  infrastructures.  The  OGC  includes  many  standards,  such  as  Observations  and 
Measurements  (O&M)  [38,39],  SensorML  [40],  Transducer  Model  Language  (TML)  [41],  Sensor 
Observation  Service  (SOS)  [42],  Sensor  Planning  Service  (SPS)  [43],  Sensor  Alert  Service  (SAS)  [44], 
and  Web  Notification  Services  (WNS)  [45]. 

One  example  system,  Agilla  [21-23],  is  a  framework  used  to  monitor  sensor  systems  connected  to  a 
sensor  network.  Agilla  uses  protocols  with  specific  conditions  that,  when  met,  will  perform  a  specific 
action  or  actions.  For  example,  the  actions  and  conditions  may  be  to  activate  other  protocols  when  a 
sensor  reports  a  temperature  above  a  specific  threshold.  The  newly  activated  protocols  may  then 
coordinate  other  sensors  to  collect  data,  invoke  algorithms  for  further  analysis,  or  even  activate  more 
protocols  to  perform  a  specific  action  or  actions.  Figure  7(a)  shows  an  Agilla  network  with  a  fire 
detection  protocol  on  one  sensor  node.  The  fire  detection  protocol  has  the  task  of  detecting  a 
temperature  above  a  specific  threshold.  Once  the  temperature  threshold  is  reached,  the  protocol  will 
activate  other  fire  detection  protocols  on  more  sensors  nodes,  Figure  7(b).  As  the  protocols  activate  on 
the  other  sensor  nodes,  the  protocols  will  determine  the  perimeter  of  the  fire  and  then  send  the 
perimeter  data  to  a  new  protocol,  Figure  7(c),  which  then  activates  fire  services  [21-23].  Another 
example  system,  Geographical  Information  System  Framework  [24,25],  leverages  several  different 
frameworks  in  the  overall  management  of  sensor  systems  and  algorithms  on  a  sensor  network  as 
depicted  in  Figure  8  The  framework  includes  knowledge  models,  such  as  ontologies,  for  describing 
sensors,  algorithms,  and  tasks.  Service-oriented  architectures  are  used  to  handle  communications 
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among  all  systems,  and  geographic  information  system  placement  logic  is  used  for  tasking.  The 
different  frameworks  operating  together  allow  end  users  or  autonomous  systems  to  query  the 
framework  for  available  sensor  systems  and  then  task  the  sensor  systems  to  retrieve  data  or  to  perform 
specific  actions  based  on  sensor  data  [24,25]. 


Figure  7.  (a)  Fire  detection  protocol  on  a  node  in  an  Agilla  sensor  network  that  detects  a 
fire;  (b)  Protocol  activates  other  protocols  on  different  nodes  to  determine  the  perimeter  of 
the  fire  and  then  activates  other  protocols  for  fire  services;  and  (c)  Activation  of  fire  services. 


Figure  8.  Geographical  Information  System  Framework  displaying  interconnections  among 
the  geographic  information  system,  service-oriented  architecture,  ontologies  and  end  user 
software  applications. 


4.  Persistence  Surveillance  Sensing  Environment 

To  demonstrate  the  efficacy  of  the  extended  ontological  problem-solving  framework,  a  persistence 
surveillance  sensing  environment  was  constructed  from  a  family  of  emulated  unattended  ground 
profiling  sensor  systems  and  algorithms.  The  profiling  sensors  provide  a  means  for  capturing  signals  of 
objects  which  pass  through  a  profiling  sensor’s  field  of  view.  The  signals  are  then  passed  to  algorithms 
which  create  profiles  of  the  signals,  which  are  then  sent  to  other  algorithms  for  further  processing,  such 
as  object  classification  or  visualization.  The  profiling  sensors  have  a  common  theme  in  that  they  are 
low  cost  and  provide  a  signal  that  can  be  classified.  The  profiling  sensors  are  denoted  by  the 
nomenclature  PFx  [46].  The  PFx  sensors  may  use  a  variety  of  sensing  bands,  including  visible,  near 
infrared,  short-wave  infrared,  mid-wave  infrared,  and  long-wave  infrared  bands.  They  typically  share 
a  common  design  principle  of  using  a  sparse  detector  array. 
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Figure  9(a)  shows  a  sparse  detector  PFx  sensor  consisting  of  sixteen  near-infrared  detectors  in  a 
vertical  deployment  with  no  relative  horizontal  displacement  and  a  reflector  pole.  When  an  object 
passes  between  the  two  poles,  which  is  the  field  of  view  of  the  sensors,  the  resulting  signal  will  be 
recorded.  An  algorithm  then  processes  the  signal  by  formatting  the  raw  sensor  data  using  run-length 
encoding.  The  formatted  sensor  data  may  be  used  by  other  algorithms  to  visualize  the  acquired  data  as 
a  silhouette  shown  in  Figure  9(b).  Other  possible  configurations  of  a  vertical  near  infrared  sparse 
detector  may  include  a  horizontal  displacement,  which  may  be  used  to  determine  the  velocity  of  an 
object  [46-53].  The  chain  of  creating  raw  sensor  data,  generating  profiles,  and  then  processing  the 
profiles  for  visualization  or  classification  provides  a  unique  opportunity  to  show  how  the  prototype 
ontological  framework  can  autonomously  assign  the  PFx  sensors  and  algorithms  to  the  subtasks  of 
various  missions  based  on  their  relationships  and  capabilities. 

Figure  9.  (a)  PFx  sensor  system  using  sixteen  near-infrared  detectors  deployed  vertically 
with  no  horizontal  displacement;  and  (b)  Silhouette  generated  by  an  algorithm  operating  on 
the  sensor  data. 

(a)  (b) 
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5.  Reasoning  Process  for  Assigning  Sensor  Systems  and  Algorithms  to  Missions 

5.1.  Problem-Solving  Framework  for  Assigning  a  Synthesis  of  Systems  to  Mission  Specifications 

To  address  the  challenge  of  assigning  sensor  systems  and  algorithms  to  high-level  missions  the 
previous  work  by  Qualls  and  Russomanno  [7]  was  extended  with  the  concept  of  a  mission  developed 
from  eliciting  knowledge  from  SMEs.  Figure  10  shows  the  original  ontology  of  the  problem 
solving-framework,  as  seen  in  Figure  3,  extended  with  an  ontology  for  describing  mission  specifications. 
The  extended  ontology  is  shown  here -with  two  additional  classes:  Mission_Sensor_System  in  gray,  and 
Mission  in  red.  The  Mission  class  has  five  supporting  classes,  also  in  red,  to  describe  mission 
specifications:  Target,  MissionJTask,  Action_Process,  Action_Condition,  and  Action jObjectjDfJnterest. 
The  primary  goal  of  the  ontology  in  the  prototype  ontological  framework  is  to  support  the  synthesis  of 
the  Mission_Sensor_System,  which  is  a  synthesis  of  systems  assigned  to  a  mission. 
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Figure  10.  Extension  of  the  prototype  ontological  problem-solving  framework  for 
matching  sensor  systems  to  algorithms  to  form  a  synthesis  of  systems  that  may  now  be 
assigned  to  subtasks  of  missions. 
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A  Mission_Sensor_System  class  describes  a  possible  combination  of  Matched _Sens or JSystem 
instances  and  a.  Mission  instance  through  the  two  object  type  properties  has_Matched_Sensor_System 
and  has_Mission.  A  Mission_Sensor_System  may  have  many  Matched_Sensor_System  instances  but 
one  only  Mission  instance.  The  Mission  class  describes  the  various  specifications  of  a  mission.  The 
Mission  class  leverages  two  other  classes,  Target  and  Mission_Task,  to  define  mission  specifications. 
The  Target  class  describes  the  object  that  the  mission  needs  to  detect,  such  as  human  or  animal.  The 
Mission_Task  class  describes  the  process  and  condition  which  must  take  place  on  the  Target  instance. 
To  define  the  process  specification,  the  Mission_Task  leverages  two  other  classes:  Action _Process  and 
Action_Condition.  The  Action _Process  class  describes  a  specification  process,  such  as  “classify”  or 
“visualize”  for  the  detected  Target  instance.  The  Action _Condition  class  describes  further 
specifications  that  the  Action_Process  might  require.  Last,  the  class  Action _Object_Of_lnterest 
describes  objects  that  a  Target  instance  may  be  associated  with,  for  example,  objects  that  may  be 
carried  by  a  human  or  animal. 

5.2.  Ontological  Framework  Rules 

The  original  prototype  ontological  problem-solving  framework  used  SPARQL  [10],  a  graph-matching 
query  language  to  implement  the  rules  to  query  the  instance  data  and  return  possible  synthesis  of 
systems.  SPARQL  rules  can  be  regarded  as  Horn  clauses  with  addition  logical  constraints  The  rules 
contain  the  following  two  components;  CONSTRUCT  and  'WHERE  clauses.  First,  the  CONSTRUCT 
clause  returns  possible  object  instances,  which  contain  new  properties,  derived  properties,  and  links  to 
other  class  instances  and  their  corresponding  attributes.  Second,  the  WHERE  clause  contains  statements 
that  specify  constraints.  The  constraints  include  the  properties  that  must  exist  and  the  logical 
constraints  that  properties  of  a  class  instance  must  satisfy  before  the  rule  will  execute.  Each  of  the 
constraints  in  a  single  rule  are  comiected  via  a  logical  conjunction  (logical  AND),  whereas  a  collection 
of  rules  of  a  common  theme  are  connected  via  a  logical  disjunction  (logical  OR).  Once  all  properties 
and  logical  constraints  of  the  WHERE  clause  are  satisfied,  the  corresponding  CONSTRUCT  clause 
will  return  the  possible  object  instance  or  instances. 
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Figure  11.  (a)  Example  SPARQL  rule  with  CONSTRUCT  and  WHERE  clauses,  which 
returns  a  possible  Mission_Sensor_System  instance;  (b)  Instance  diagram  of  CONSTRUCT 
clause;  and  (c)  Instance  diagram  of  WHERE  clause. 
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New  rules  were  developed  for  assigning  the  synthesis  of  systems  to  missions,  thus  returning 
possible  MissionJSensorJSystem  instances.  Figure  11  shows  an  example  SPARQL  rule  that  quenes  the 
existing  instance  data  and  returns  a  Mission_Sensor_System  instance  in  the  CONSTRUCT  clause  when 
the  properties  aid  logical  constraints  are  satisfied  in  the  WHERE  clause.  The  CONSTRUCT  clause 
in  Figure  11(a)  contains  three  statements.  The  first  statement  declares  the  object  instance 
Instance_Mission_Sensor_System  to  be  of  class  type  Mission_Sensor_System.  The  second  two 
statements  establish  links  to  the  possible  existing  instances  through  the  two  properties:  hasjvlission 
and  has_Matched_Sensor_System.  To  establish  these  two  links,  the  two  properties  are  linked  to  two 
variables  InstanceJMission  aid  Instance_Matched_Sensor_System,  respectively.  The  WHERE  clause 
in  Figure  11(b)  contains  five  statements.  In  the  first  two  statements,  the  object  variable 
histaiceJ\4atched_Sensor_System  is  instantiated  with  an  instance  of  class  type  Matched_Sensor_System 
aid  the  variable  Matched_Process_Type  is  instantiated  with  the  value  from  the  data  type  property 
has_Process_Type  from  the  sam t  Matched_Sensor_System  instance.  In  the  second  two  statements,  the 
object  variable  Instance_Mission  is  instantiated  with  an  instance  of  class  type  Mission  and  the  variable 
Mission_Process_Type  is  instaitiated  with  the  value  from  the  data  type  property  has_Process_Type 
from  the  sane  Mission  instance.  Tie  final  statement  in  the  WHERE  clause  contains  the  FILTER 
command  which  appears  as  a  simple  logical  constraint  that  compares  two  variables.  This 
particula  FILTER  command  compares  the  two  data  type  variables  Matched_Process_Type  aid 
Mission_Process_Type  for  equality.  When  the  inference  engine  processes  this  rule,  the  CONSTRUCT 
clause  will  return  a  possible  Mission_Sensor_System  instance  with  links  to  a  Matched _Sensor _System 
instance  and  links  to  an  assigned  Mission  instance  if  the  two  instances  and  properties  exist  and  if  the 
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two  properties  are  equal  in  the  WHERE  clause.  Once  a  Mission  JSensorJSystem  instance  has  been 
returned  by  the  rules,  the  ontological  framework  can  then  execute  the  mission  by  coordinating  all 
synthesized  systems,  sensors,  and  algorithms  assigned  to  that  mission  and  returning  the  results  of  the 
mission  via  a  procedural  attachment  statement. 

One  note  of  interest  is  the  subsumption  qualities  of  the  logical  constraint  for  the  FILTER  command. 
For  example,  if  the  Matched_Process_Type  variable  is  set  to  a  subclass  of  the  Process_Type  and  the 
Mission_Process_Type  variable  is  set  to  a  superclass  of  the  Process_Type,  the  rule  will  need  to  return 
true  or  false  depending  on  a  set  threshold  for  the  semantic  distance  between  the  variables.  To  set  this 
threshold  the  authors  decided  to  set  the  mission  as  a  priority.  The  reason  for  this  selection  is  based  on 
feedback  from  the  SMEs  in  tenns  of  CONOPs.  For  example,  a  mission  may  be  created  for  a  specific 
target  but  no  synthesis  of  systems  can  complete  that  exact  mission.  Forcing  the  framework  to  assign 
systems  only  to  exact  matches  of  missions  would  severely  limit  the  capabilities  of  the  framework.  So 
the  authors  decided  to  allow  the  framework  to  return  possible  best  matches  between  a  synthesis  and  a 
mission.  For  the  ontological  framework  the  semantic  distance  threshold  has  been  set  as  follows  for  the 
prototype.  There  are  two  basic  conditions:  if  the  property  of  the  mission  is  a  subclass  of  the  matched 
system  or  if  the  property  of  the  mission  is  a  superclass  of  the  match  system.  First,  if  a  mission  property 
such  as  has_Action_Object_Of_Interest  is  set  to  a  value  that  is  a  subclass  of  the  same  property  of  the 
matched  systems,  the  framework  will  assign  the  matched  system  to  the  mission  up  to  the  top-level 
superclass  that  property  may  have.  For  example,  if  the  mission  has_Action_Object_Of_Interest 
variable  is  set  to  pistol,  the  framework  would  assign  matched  systems  up  to  highest  class  domain  of  the 
property  in  this  case  has_Action_Object_Of_Interest,  which  is  the  class  Object_Of_Interest.  Second,  if 
the  has_Action  Object  Ofylnterest  variable  of  the  matched  system  is  set  to  a  value  that  is  a  subclass 
of  the  same  property  of  the  mission  then  an  assignment  will  take  place.  For  example,  if  the  mission 
has_Action_Object_Of_Interest  property  is  set  to  pistol,  the  framework  would  assign  matched  systems 
that  are  subclasses  of  pistol. 

The  rules  m  the  ontological  problem-solving  framework  all  follow  a  similar  structure  outlined  in 
Figure  11.  The  rules  bind  on  all  combinations  of  Mission  and  Matched  JSensorJSystem  instances  and 
return  possible Mission_Semor_System  instances  in  the  CONSTRUCT  clause  when  the  corresponding 
properties  exist  and  logical  constraint  statements  are  met  in  the  WHERE  clause.  Figures  12  and  13 
each  show  one  of  many  different  kinds  of  rules  that  return  possible  Mission  JSensorJSystem  instances 
and  their  resulting  instance  diagrams.  These  rules  bind  on  properties  of  the  Matched  JSensorJSystem 
instance  that  link  back  to  other  instances,  such  as  the  type  of  process  the  system  can  accomplish,  and 
additional  properties,  such  as  conditions  on  the  process  that  may  or  may  not  be  optional.  The  rules  also 
bind  on  properties  of  a  Mission  instance,  which,  as  previously  discussed,  include  Target , 
MissionJTask,  Action Process ,  and  Action  ^Condition .  Figure  12  shows  a  rule  which  binds  on  a  simple 
mission  to  process  a  target  with  no  conditions,  such  as  “classify  human  male”.  Figure  13  shows  a  rule 
that  binds  on  more  advanced  missions  that  processes  a  target  with  conditions,  such  as  “visualize  horse 
carrying  backpack”  or  “classify  human  male  with  height  greater  than  six  feet”,  respectively. 
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Figure  12.  Rule  and  instance  diagram  showing  how  a.Mission_Sensor_System  is  returned 
for  a  simple  Mission  if  a  Matched _Sensor_System  can  accomplish  the  mission  based  on 
Action_Process  and  Target  properties. 
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Figure  13.  Rule  and  instance  diagram  showmg  how  a Mission_SensorJ$ystem  is  returned 
for  an  advanced  Mission  if  ^Matched _Sensor _System  can  accomplish  the  mission  based  on 
Action  Process^ActionjOondition ,  and  Target  properties. 
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Figure  13.  Cont. 
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6.  Example  of  Assigning  a  Synthesis  of  Systems  to  Mission  Specifications 

To  show  how  the  prototype  ontological  problem-solving  framework  operates,  a  small  example  has 
been  created.  Figure  14  shows  an  overview  of  all  emulated  assets  instantiated  and  the  resulting 
synthesis  of  systems  and  assigmnent  to  a  mission  instance.  To  begin,  the  ontological  framework  was 
instantiated  with:  (i)  one  emulated  sensor  systems  Photo ^Conductive  (Figure  14(a));  (ii)  two 
algorithms,  Pixel ^Extract or  (Figure  14(b))  and  Naive _Bayes_Classifier  (Figure  14(c));  and  (lii)  and 
one  mission  instance  (Figure  14(d)).  The  following  section  will  detail  how  the  sensor  system  and 
algorithms  are  matched  together  to  form  a  synthesis  of  systems  that  are  assigned  to  a  mission.  Each  of 
the  instances  has  many  different  data-type  properties,  but  for  this  example  only  a  few  relevant 
properties  are  show  in  Figure  14. 

The  Sensor  instance  Photo _Conductive  has  four  properties;  has  ^Horizontal  JPixel  Resolution  set 
to  640  pixels,  hasJVferticalJPixelJR.esolution  set  to  480  pixels,  has_Horizontal__Detector_Displacement, 
and  has_Vertical_Detector_Displacement  both  set  to  none.  The  Photo _Conductive  instance  represents 
a  sensor  capable  of  generating  a  signal  profile  of  a  passing  target.  The  Algorithm  instance 
Pixel JExtr actor  has  three  properties;  has_Input_Horizontal_Re solution  set  to  640  pixels, 
hasJbiput_Vertical_Resolution  set  to  480  pixels,  and  has_Output_Data_Type  set  to  image.  The 
Pixel  Extractor  instance  represents  an  algorithm  capable  of  loading  a  raw  signal  profile  data 
in  640  x  480  fonnat  and  then  generating  a  formatted  signal  profile  into  an  image  format.  The  second 
Algorithm  instance  Naive _Bayes_Classifier  has  three  properties:  (i)  has_In putD ata_T ype  set  to 
image;  (ii)  has_Classification_Target  set  to  human  male;  and  (iii)  and  has_Process_Type  set  to 
classify.  The  Naive JBayes_Classifler  instance  describes  a  classifier  that  operates  on  features  of  an 
image  and  then  classifies  the  image  as  a  human  male  or  not  a  human  male. 

The  Mission  instance  represents  a  mission  that  requires  the  detection  of  human  males,  i.e.9  classify 
human  male.  The  Mission  instance  has  two  object-type  properties,  has_Target  and  has_Mission_Task, 
which  link  to  the  Target  instance  and  Mission_Task  instance.  The  Target  instance  describes  a  human 
male  instance  that  has  many  properties,  such  as  has_Name  and  not  shown  hasJHeight,  and  has_Weight. 
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The  instance  MissionJTask  has  two  object-type  properties  has_Action_Process  which  links  to  the 
instance  ActioijProcess  “classify”  and  the  property  has_Action_Condition  which  links  to  the 
ActionjCondition  instance  “none”.  The  Action_Process  instance  has  many  data-type  properties,  such 
as  has_Process_Type,  which  can  have  the  values  classify,  profile  generator,  convertor,  and  visualizer. 
For  this  case,  the  data-type  property  is  set  to  classify.  The  ActionjCondition  instance 
“none”  has  three  data-type  properties,  has_Condition_Type,  has_ConditionJProperty,  and 
has__Condition_Value,  each  set  to  “none”  and  one  object-type  property,  has_Condition_Object,  which 
links  to  the  instance  Action_Object_Of_lnterest  “none”.  The  instance  Action_Object_OfyInterest 
“none”  is  of  type  Object  jDf_lnte rest  which  describes  a  possible  object  the  Target  may  be  holding  or 
wearing,  but  in  this  example,  the  mission  does  not  specify  if  the  human  male  is  carrying  an  object,  so 
all  values  are  set  to  none. 

Figure  14.  Mission_Sensor_System  instance  diagram  showing  a  linked  Mission  instance 
“classify  human  male”  matched  to  a  synthesized  system  capable  of  satisfying  the 
high-level  mission,  (a)  Sensor  instance  Photo _Conductive\  (b)  Algorithm  instance 
PixelJLxtr actor ;  (c)  Algorithm  instance  Naive JBayesjClassifie r;  (d)  Mission  instance 
“classify  human  male”;  (e)  ProJUmg_Sensor_System  instance;  (f)  Matched _Sensor _System 
instance;  and  (g )  Mission_Sensor_System  instance. 
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With  all  systems  and  a  mission  instantiated  on  the  prototype  ontological  framework,  rules  such  as 
those  in  Figures  12  and  13  will  process  the  instance  data  to  form  a  synthesis  of  systems  and  assign  the 
synthesis  to  the  mission.  The  first  synthesis  of  systems  to  be  returned  is  a  Profihng_Sensor_System 
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instance  shown  in  Figure  14(e).  The  Profiling _Sensor_Systein  instance  was  returned  because  the 
properties  of  Photo  jConductive  and  Pixel JLxfractor  matched,  i.e.9  the  output  pixel  resolutions  of  the 
Photo JConductive  and  input  pixel  resolutions  of  the  PixelJLxtr actor  matched  to  640  x  480.  The 
synthesized  Profiling_SensorJSystem  instance  contains  two  derived  object-type  properties  that  link  to 
the  Sensor  instance  Ph oto  JConductive  and  the  Algorithm  instance  Pixel JExfract  or ,  called  hasJSensor 
and  has_Algorithm.  The  Profiling_Sensor_System  instance  represents  a  synthesis  of  systems  capable 
of  formatting  a  raw  signal  profile  into  a  formatted  ’'image"  profile. 

The  next  pass  of  the  inference  cycle  will  produce  the  second  synthesis  of  systems;  the 
Matched _Sensor _Sy stem  instance  shown  in  Figure  14(f).  The  Matched _Sensor _System  instance 
contains  the  two  object-type  properties  has_Profiling_Sensor_System,  which  links  to  the  synthesized 
Profiling_Sensor_System  instance,  and  has_Algorithm,  which  links  to  the  Naive JBayesjClassifier 
instance.  Tie  algorithm  Naive  JBayesjClassifier  was  matched  to  the  Profiling_Sensor_System  instance 
because  the  data-type  property  has_Output_Data_Type  set  to  “image”  matched  the  data-type  property 
has_Input_Data_Type  set  to  “image”,  respectively.  Tie  new  Matched _Sensor _System  instance 
represents  a  synthesized  system,  which  generates  raw  signal  data  that  can  then  be  classified  as  a  human 
male  or  not  a  human  male. 

On  the  next  inference  cycle,  the  rules  return  a  possible  Mission_Sensor_System  instance,  shown  in 
Figure  14(g),  which  assigns  the  synthesized  system  Matched _Sensor _Sy stem  instance  to  the  simple 
Mission  instance  because  of  two  sets  of  properties.  First,  the  data-type  property 
has_Classification_Target  value  “human  male”,  which  is  linked  to  the  Matched  Sensor _Sy stem 
through  the  has_Algorithm  object-type  property,  matches  to  the  data  type  property  has_Name  “human 
male”  in  the  Target  instance,  which  is  linked  to  the  instance  Mission  through  the  object-type  property 
has_Target.  Second,  the  Naive  JBayesjClassifier  instance  has  the  data-type  property  has_Process_Type 
set  to  the  value  “classify”.  The  Naive _Bayes_Classi filer  instance  is  linked  to  the  Matched_Sensor_ 
System  instance  through  the  object-type  property  has_Algorithm  because  the  data-type  property 
has_Process_Type  of  the  instance  Action_Process  is  set  to  “classify”.  Action_Process  is  linked  to  the 
instance  Mission_Task  through  the  object-type  property  has_Action_Process,  which  in  turn  is  linked  to 
the  Mission  instance  through  the  object-type  property  has_Mission_Task.  The  synthesized 
Mission_Sensor_System  instance  links  to  the  synthesized  Matched _Sensor_Sy stem  instance  through 
the  object-type  property  has_Matched_Sensor_System  and  links  to  the  Mission  instance  through  the 
object-type  property  has_Mission  and  represent  synthesized  systems  ready  to  be  coordinated  to 
complete  the  mission  classify  human  male.  The  returned  Mission_Sensor_System  system  is  added  as 
an  instance  in  the  ontology  so  further  inference  can  leverage  the  synthesis  of  systems  and  mission  for 
further  complex  mission  tasking  or  for  actual  coordination  to  execute  the  mission.  Although  Figure  14 
shows  relatively  simple  properties,  and  the  rules  in  Figure  12  and  Figure  13  bind  on  simple 
compatibility  constraints,  further  properties  and  more  complex  uses  of  the  SPARQL,  FILTER,  and 
OPTIONAL  commands  may  allow  for  more  complex  synthesized  systems  to  be  returned  and  assigned 
to  increasingly  sophisticated  missions. 
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7.  Instantiated  Emulated  Profiling  Sensor  Systems  and  Algorithms 

To  show  the  efficacy  of  the  ontological  problem-solving  framework,  several  emulated  profiling 
sensor  systems  and  algorithms  were  instantiated  as  complete  MatchedJSensorJSystem  instances  in  the 
ontological  problem-solving  framework  as  a  prototype  environment  for  testing.  In  the  prototype 
environment,  nine  different  Mission  instances  and  six  different  MatchedJSensorJSystem  instances 
were  instantiated,  Figure  15.  Each  of  the  various  Matched _Sensor_Sy stem  instances  contained 
Profiling JSensor^System  instances  made  up  of  matched  emulated  sensor  systems  and  algorithms,  with 
some  of  the  emulated  systems  shared  between  different  Matched _Sensor _Sy stem  instances. 


Figure  15.  Instantiated  examples  on  the  ontological  framework:  (a)  Nine  different  Mission 
instances  consisting  of  detection  and  classification  of  targets  and  visualization 
of  targets;  and  (b)  Six  different  MatchedJSensorJSystem  instances  with  links  to 
Profling_Sensor_Systemy  Sensor  and  Algorithm  instances  matched  together  to  form  a 
synthesized  system  capable  of  performing  a  task. 

(a)  Mission  Instance  Data  (b)  Matched_Sensor_System  Instance  Data 


Mission :  classify  human 


Mission:  classify  human  carrying  backpack 


Mission:  classify  human  carrying  sub-machine  gun 


Mission:  classify  human  with  height  greater  than  six  feet 


Mission:  classify  animal 


Mission:  visualize  human  carrying  backpack 


Mis sion:  visualize  human  carrying  sub-machine  gun 


Mission:  visualize  animal  carrying  weapon 


Mission:  visualize  human 


Matched: 

PFt  system 

Target: 

human,  animal 

Action^Process: 

classify 

Action_Condition: 

no  conditions 

Matched:  PF2  system 

Target:  human 

Action_Process:  classify 
Action_Condition:  with  object  of  interest 


Matched: 

PF3  system 

Target: 

human 

Action_Process: 

classify 

Action_Condition: 

with  weapon 

Matched: 

PF4  system 

Target: 

any  target 

Action_Process: 

classify 

Action_Condition: 

height  above  six  feet 

Matched:  PF6  system 

Target:  human 

Action_Process:  visualize 
Action_Condition:  with  object  of  interest 


Matched: 

PF6  system 

Target: 

any  target 

Action_Process: 

visualize 

Action_Condition: 

with  weapon 

When  the  ontological  problem-solving  framework  begins,  the  inference  cycle  processes  the  rules 
similar  to  those  in  Figures  12  and  13.  When  the  inference  cycles  terminate,  sixteen  new 
Mission  JSensorJSystem  instances  were  returned  as  shown  in  Figure  16.  From  Figure  16,  multiple 
Matched_Sensor_System  instances  were  matched  to  a  single  Mission  instance  while  in  some  cases  a 
single  MatchedJSensorJSystem  instance  was  matched  to  multiple  Mission  instances.  For  example,  the 
Mission  instance  “classify  human  carrying  sub-machine  gun”  can  be  completed  by  two  different 
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Matched JSensor _Sy stem  instances:  “PF2  system”  and  “PF3  system”  because  the  Action_Process 
“classify”  of  the  Mission  matched  the  Process_Type  “classify”  of  both  PFx  systems  and  each  of  the 
Mission  instance  Action_Obj ect_Of_Interest  “sub-machine  gun”  matched  the  Action_Condition  of 
both  PFx  systems.  This  particular  assignment  also  represents  how  the  authors  chose  to  handle 
subsumption  with  a  semantic  distance  threshold  in  that  the  Action_Condition  of  both  PFx  systems 
were  not  “sub-machine  gun”  but  “object  of  interest”  and  “weapon”,  which  are  each  super  classes  of 
“sub-machine  gun”.  Assigning  matched  systems  in  this  way  allows  the  ontological  framework  to  “best 
fit”  a  mission  to  a  synthesis  of  systems.  Also  the  Matched _Sensor_System  instance  “PF5  system”  can 
complete  three  different  Mission  instances:  “visualize  human  carrying  backpack”,  “visualize  human 
carrying  sub-machine  gun”,  and  “visualize  human”. 

Once  the  assignments  have  been  completed,  z.e.,  the  Mission JSensorJSystem  instances  have  been 
returned  by  the  inference  engine,  the  prototype  ontological  framework  selects  a  single  completed 
Mission  JSensorJSystem  instance  through  a  rule  and  then  coordinates  all  sensor  systems  and  algorithms 
associated  with  the  instance  via  a  procedural  attachment  within  the  rule  to  complete  the  mission  and 
return  the  results.  Once  the  mission  has  been  completed,  the  ontological  framework  will  then  select  the 
next  Mission_Sensor_System  instance  to  coordinate,  complete,  and  return  the  results.  Since  the 
ontological  problem-solving  framework  is  in  a  laboratory  prototype  stage,  only  a  single  mission  is 
completed  at  a  time.  Also,  if  the  Mission  instance  is  matched  to  several  Matched  JSensorJSystem 
instances,  the  same  mission  will  be  completed  for  each  assignment  regardless  if  it  was  previously 
completed.  Improved  systems  can  be  developed  that  allow  for  simultaneous  mission  coordination, 
completion,  and  avoidance  of  repeating  a  mission,  but  the  focus  here  is  on  proof-of-concept 


Figure  16.  Sixteen  new  MissionJSensorJSystem  instances  were  returned  with  derived 
relationships  after  the  inference  cycle  completed. 


Matched:  PF2  system 

Target:  human 

Action_Process:  classify 
Action_Condition:  with  object  of  interest 


Mission :  classify  human 
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Figure  16.  Cont. 


Mission :  classify  human 


Matched :  PF^  system 

Target:  any  target 

Action_Process:  classify 
Action_Condition:  height  above  six  feet 


Mission:  classify  human  carrying  sub-machine  gun 


Matched:  PF2  system 

Target:  human 

Actio  n_Process:  classify 
Action_Condition:  with  object  of  interest 


Mission ;  classify  animal 
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Action_Process:  classify 
Action  Condition:  no  conditions 
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PF4  system 

Target: 

any  target 

Actio  n_Process: 
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Action_Condition: 

height  above  six  feet 
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Figure  16.  Cont. 


Mission :  visualize  human  carrying  backpack 


Matched :  PF6  system 

Target’  human 

Adion_Process:  visualize 
Adion_Condition:  with  objed  of  interest 
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Action_Condition  with  object  of  interest 


Mission:  visualize  human 


Matched :  PF8  system 

Target:  any  target 

Action_Process:  visualize 
Action_Condition:  with  weapon 


8.  Discussion 

The  challenge  for  the  ontological  problem-solving  framework  was  to  assign  a  synthesis  of  systems 
to  subtasks  of  mission  specifications.  Even  though  the  rules  described  in  this  paper  contain  relatively 
simple  compatibility  constraints  among  sensors,  algorithms,  and  missions,  these  rules  illustrate  an 
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important  proof-of-concept.  Namely,  a  problem-solving  approach  to  matching  Sensor  and  Algorithm 
instances  to  fonn  synthesized  Matched_Sensor_System  and  Pvofiling_Sensor_System  instances  which 
are  then  assigned  to  high-level  Mission  instances  to  form  a  Mission_Sensor_System  instance  that  is 
ready  to  be  executed  by  the  ontological  framework  or  other  autonomous  systems  for  mission 
completion.  It  is  important  to  note  that  multiple  Matched_Sensor_System  instances  were  reused  and 
assigned  to  different  Mission  instances,  which  the  Matched _Sensor jSystem  instances  were  capable  of 
satisfying.  For  example,  the  Matched  Sensor  JSystem  PF5  instance,  which  is  capable  of  visualizing  a 
human  carrying  an  object,  was  assigned  to  three  different  Mission  instances  that  required  the  detection 
of  “humans”  with  and  without  various  objects,  such  as  “weapons”  or  “backpacks”.  It  is  important  to 
realize  that  the  Mission_Sensor_System  is  more  than  just  sensors  and  algorithms  assigned  to  a  mission, 
the  Mi  ssi  on JSens  or  JSystem  is  a  synthesized  system,  which  is  capable  of  performing  the  assigned 
subtasks  to  satisfy  the  overall  mission  specification  and  returning  results  for  further  analysis  or  more 
complex  missions. 

Rules  in  the  ontological  framework  may  operate  on  more  than  just  properties  of  the  various 
instances.  For  example,  more  complex  rales  may  determine  that  certain  Mission_Sensor_System 
instances  may  be  composited  together  to  satisfy  more  complex  mission  specifications.  Possible 
complex  Mission  instances  may  include  the  detection  of  multiple  targets  and  the  tasking  of  other 
complex  synthesized  systems  to  monitor  the  targets  for  a  specific  time,  which  could  be  represented  as  a 
single  MissionJSensor JSystem  instance.  Other  rales  may  even  generate  new  missions  or  decompose 
missions  into  specifications  for  subtask  assigmnent.  Without  leveraging  ontologies,  rales,  the  inference 
engine,  and  the  concept  of  synthesized  systems,  all  of  the  sensors  and  algorithms  would  need  to  be 
configured  a  priori  for  the  anticipated  missions  and  reconfigured  for  unanticipated  missions.  Most 
missions  and  subtasks  are  not  known  at  the  time  of  system  deployment,  therefore,  a 
problem-solving  approach  may  opportunistically  assign  synthesized  systems  to  subtasks  of  high-level 
missions  in  real-time,  which  is  an  extremely  important  capability  for  dynamically  changing 
requirements  in  a  particular  environment. 

As  discussed  in  the  related  work,  many  framework  and  middleware  systems  have  been  researched 
and  developed  to  assign  systems  to  missions.  Some  of  the  middleware  systems  used  a  priori  matching 
of  assets  to  mission  tasks,  such  as  Agilla  [21-23],  which  limits  the  reuse  of  assets  for  other  tasks 
without  new  matching  occurring  a  priori.  Other  systems  that  use  knowledge  bases  for  sensor  mission 
assigmnent  [12-19]  leveraged  ontologies  and  other  techniques  for  automated  sensor  mission 
assignment.  As  stated  previously,  the  reasoning  and  use  of  knowledge  engineering  techniques  by  the 
authors  for  the  prototype  ontological  framework  is  similar  to  other  efforts  in  some  aspects.  The  work 
described  in  this  paper  differs  from  these  other  works  in  that  the  domain  of  tire  missions  and  assets 
were  limited  to  a  persistence  surveillance  sensing  environment.  By  limiting  the  domain,  the  research  of 
the  prototype  ontological  problem-solving  framework  could  focus  on  providing  a  complete  solution 
that  not  only  assigns  assets  to  missions  but  also  includes  a  coordination  system  that  connects  to 
emulated  assets  and  completes  the  mission. 

Although  the  priority  at  this  stage  of  this  research  is  the  logical  problem-solving  framework, 
another  important  aspect  is  performance.  Performance  can  be  analyzed  along  several  dimensions, 
including  scale-up  analysis  with  solution  finding,  mission  operation  time,  and  mission  completion 
rates.  First,  scale-up  perfonnance  analysis  is  limited  at  this  point,  but  the  ontological  framework  can 
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scale  to  a  very  large  number  of  instantiated  sensors,  algorithms  and  missions,  limited  only  by  physical 
memory  constraints.  The  reasoning  strategy  used  by  the  inference  engine,  along  with  the  features 
expressible  in  the  knowledge  representation  language,  dictate  the  overall  computational  complexity, 
which  in  turn  determines  the  time  for  the  ontological  framework  to  infer  all  combinations  of  sensors, 
algorithms,  and  missions.  Performance  can  be  increased  by  enabling  the  inference  engine  to  check 
multiple  sensors,  algorithms,  and  missions  in  parallel  or  by  invoking  the  inference  engine  multiple 
times  in  parallel,  while  having  a  strategy  in  place  to  eliminate  redundant  bindings. 

Currently,  the  prototype  ontological  framework  has  been  tested  with  fifty  systems  not  detailed  in 
tins  paper.  The  ontological  framework  takes  less  than  two  seconds  to  find  all  possible  combinations, 
which  equates  to  over  500  new  combinations  after  the  inference  cycle  completes.  As  more  sensor 
systems  and  algorithms  are  added  to  the  ontological  framework,  combinatorial  explosion  becomes  an 
issue.  Combinatorial  explosion  may  be  somewhat  mitigated  once  asset  resource  control  and  time 
constraints  are  taken  into  account  in  the  rules.  First,  as  the  number  of  systems  increase,  new  systems 
for  the  ontological  framework  will  need  to  be  researched  and  developed  that  limit  the  number  of 
solutions  found  and  to  determine  the  correctness  of  the  proposed  solution.  Second,  mission  operation 
time  is  determined  by  how  the  ontological  framework  or  other  systems  can  execute  and  complete 
missions.  To  increase  performance,  the  framework  needs  to  operate  missions  in  parallel  and  prioritize 
matched  sensors  and  algorithms,  which  are  assigned  to  different  missions.  Third,  the  prototype 
ontological  framework  described  in  this  paper  does  not  take  into  account  competing  missions,  i.e., 
resource  management.  New  research  is  focusing  on  designing  systems  that  provide  information  to  the 
ontological  framework  for  priority  of  missions,  such  as  time  constraints,  availability  of  sensor  systems 
and  algorithms,  i.e.,  resource  control  and  time  difference  between  collection  of  data  and  mission  time 
completion.  Other  mechanisms  will  need  to  be  established  that  prevent  a  mission  from  never 
completing.  For  example,  a  mission  may  be  to  classify  humans,  but  humans  may  never  be  detected 
thus  locking  the  resources  for  that  mission  indefinitely.  Placing  time  constraints  on  active  missions 
may  prevent  the  never  ending  mission. 

9.  Conclusions 

Although  the  paper  only  shows  PFx  sensors,  algorithms,  and  missions  related  to  operation  of  those 
sensors  and  algorithms  instantiated  on  the  ontological  framework,  the  principles  and  techniques  that 
have  been  demonstrated  may  be  appropriate  for  other  types  of  sensors,  algorithms,  or  missions. 
Development  and  deployment  of  new  sensor  systems  and  algorithms  will  continue  to  create 
challenges,  such  as  discovering  appropriate  sensor  systems  and  algorithms  to  satisfy  tasks  which  may 
then  be  assigned  to  subtasks  of  mission  specifications.  The  lack  of  explicit  knowledge  models  used  to 
describe  the  capabilities  of  sensor  systems  and  algorithms  and  the  specifications  on  high-level  missions 
compounds  the  challenge  even  further.  To  allow  the  flexibility  of  assigning  systems  to  unanticipated 
missions,  the  framework  must  leverage  knowledge  models,  such  as  ontologies,  rules,  and  inference 
engines,  in  a  machine-interpretable  format  to  perform  automated  synthesis  and  assignment  of  sensor 
systems  and  algorithms.  The  use  of  ontologies  facilitates  inference  with  rules  allowing  the  prototype 
ontological  problem-solving  framework  to  autonomously  reason  about  how  a  synthesis  of  systems  may 
be  fonned  and  then  assigned  to  missions.  New  research  for  the  prototype  is  focusing  on  addressing  the 
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issues  raised  in  the  discussion  section,  such  as  combinatorial  explosion,  resource  constraints,  mission 
completion  time,  and  other  areas.  The  problem-solving  approach  developed  in  this  paper  for  the 
laboratory  prototype  ontological  framework  is  the  first  step  towards  achieving  reuse  of  systems 
without  an  a  priori  configuration,  flexible  assignment  of  synthesized  systems  to  mission  subtasks 
through  automated  inference,  and  addressing  further  issues  affecting  a  frameworks  ability  to 
autonomously  coordinate  assets. 
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ABSTRACT 

Remote  detection  of  harmful  intent  is  necessary  for  effective  and  appropriate  countermeasures  and  will  reduce 
risks  to  life  and  property.  Trained  human  observers  and  sensor  systems  typically  use  facial  expressions,  gaits, 
gestures,  perspiration,  and  a  number  of  other  observable  characteristics  as  possible  indicators  of  harmful  intent 
with  mixed  results.  It  is  proposed  that  responses  of  human  subjects  to  external  stimuli  can  be  used  as  additional 
indicators  of  harmful  intent  in  surveillance  contexts.  A  variety  of  alerting  stimuli,  possible  responses  to  the 
'  stimuli,  features  to  be  sensed  by  sensors,  and  the  utility  of  these  sensed  features  as  indicators  of  harmful  intent 
are  discussed  in  this  paper.  An  outology-based  data-to-decision  framework  for  assessing  human  intent,  which 
would  leverage  the  formal  representations  of  the  alerting  stimuli,  as  well  as  the  variety  of  possible  responses,  is 
proposed  in  the  context  of  Semantic  Web  infrastructure. 

Keywords:  Intent  Detection,  Data-to-Decision  framework,  Surveillance,  Human  Behavior,  Ontology. 

1.  INTRODUCTION 

The  ability  to  remotely  detect  the  covert  intent  of  humans  is  very  useful  from  the  perspective  of  security. 
According  to  a  report  by  the  Army  Research  Laboratory,1  such  a  capability  will  greatly  mitigate  the  rislcs 
involved  in  military  surveillance,  especially  in  hostile  environments,  and  enable  the  adoption  of  appropriate  and 
timely  countermeasures  to  tlireatening  situations.  These  capabilities  and  benefits  can  extend  to  non-military 
security  applications  as  well,  including  intelligent  border-monitoring  applications.  Video  footage  from  monitored 
areas,  ranging  from  border  checkpoints  to  shopping  malls,  can  be  used  in  a  preventive  capacity  in  addition  to  its 
current  primary  usage  for  post-incident  investigations.  Automated  intent  detection  from  \ndeo  surveillance,  as 
well  as  from  the  percepts  sensed  from  other  sensors  will  potentially  neutralize  the  deleterious  effects  of  human 
observer  fatigue.  Research  studies  show  even  the  attention  of  well-trained  security  guards  drops  to  unacceptable 
levels  after  only  20  minutes  of  continuously  viewing  and  analyzing  video  footage.2 

A  number  of  approaches  to  human  intent  detection,  using  trained  human  observers,  as  well  as  tracking 
methods  in  ’rideo  cameras,  have  been  adopted  at  various  monitored  areas,  such  as  airports  and  border  checkpoints 
with  mixed  success.  These  include  scrutinizing  facial  expressions,  gaits,  movements,  and  gestures.3,4  At  a  basic 
level,  intent  detection  approaches  involve  modeling  and  classification  of  tracked  human  movement  or  expression 
with  certain  rules.  More  advanced  approaches  involve  the  use  of  thermal  and  infrared  imagers  that  measure 
pulse  and  heart  rates,  or  even  display  patterns  of  blood  circulation  and  body  temperature.  The  effectiveness  of 
these  techniques  as  indicators  of  hostile  intent  is  not  well  established  at  this  time.5,6 

Tins  paper  investigates  the  use  of  a  novel  data-to-decision  framework  for  intent  detection,  using  the  responses 
of  human  subjects  to  an  alerting  stimulus.  The  responses  of  the  subjects  to  the  stimulus  can  be  used  as 
indicators  of  harmful  intent.  Consider  the  data  fusion  model  proposed  by  the  Joint  Directors  of  Laboratories 
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(JDL),7  augmented  with  a  knowledge  management  component  in  Figure  l.8  These  six  fusion  levels  cover  both 
"automated"  and  "human  in  the  loop"  processing  of  data  and  knowledge.  The  model  can  be  used  to  frame  the 
concept  of  autonomous  algorithms  and  human  users  contributing  to  an  evolving  solution  state  in  which  fused 
information  may  enable  the  identification  and  assessment  of  strategies  and  tactics  for  counterintelligence8"10 
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Figure  1.  Six  levels  of  the  data  fusion  model  augmented  with  a  data  and  knowledge  management  system. 


2.  SURVEILLANCE  AND  INTENT  DETECTION 

Stimuli  can  be  synthesized  and  introduced  into  a  sensed  environment  to  assess  the  reaction  of  humans.  Depending 
of  the  background  context  and  environment,  a  human’s  reaction  to  the  stimuli  may  be  supportive  information 
that  can  incrementally  contribute  to  an  evolving  hypothesis  about  the  intent  of  the  humans.  Audible  stimuli, 
such  as  a  recorded  human  voice  reading  out  a  message,  or  a  sound  such  as  the  breaking  of  a  twig,  or  the  cocking 
of  a  bolt-action  rifle,  may  be  used.  Flashing  lights,  a  swiveling  lamp  turning  on  and  tracking  the  movement  of 
the  subject (s),  or  even  an  increase  in  illumination  intensity  are  examples  of  visual  stimuli,  which  may  be  more 
effective  at  night,  The  choice  of  stimulus  may  be  context  dependent  as  well.  For  example,  some  stimuli,  such 
as  the  sound  of  a  rifle  being  chambered  may  be  more  appropriate  in  a  combat  zone.  The.  devices  that  generate, 
stimuli  are  generally  inexpensive  and  easily  camouflaged.  However,  they  may  be  effective  in  altering  the  behavior 
of  the  human  subjects  in  many  cases,  eliciting  a  variety  of  responses  from  individuals  and  groups,  which  may  be 
useful  in  assessing  their  intent.  Let  us  examine  each  of  the  potential  stimuli  separately. 

2.1  Alert  Stimuli  and  Responses  of  Individuals 

Suicide  bombers  axe  an  extreme  example  of  the  threat  posed  by  individuals  in  military  installations,  airports, 
and  other  public  places.  Every  year,  security  officers  trained  in  profiling  passengers  on  the  basis  of  their  general 
behavior  and  body  language  detain  thousands  of  passengers  for  further  screening  and  questioning  at  airports 
around  the  U.S.  and  abroad.  While  arrests  and  detainment  are  relatively  rare,  most  of  these  arrests  are  for  prior 
convictions  that  are  revealed  by  background  checks  and  not  because  of  any  identified  hostile  intent.5  Similarly, 
security  personnel  maiming  secured  areas,  such  as  military  bases  around  the  world,  look  for  telltale  signs  in 
human  subjects. 

Given  the  high  stakes  for  suicide  bombers  and  the  high  levels  of  anxiety,  alert  stimuli  may  be  particularly 
helpful  in  exposing  their  intentions.  Consider  the  possible  responses  of  a  suicide  bomber  to  an  alert  message  or 
to  flashing  lights  at  night,  which  may  be  triggered  manually  by  personnel  who  observe  suspicious  behavior.  A 
trained  terrorist  willing  to  die  for  his  cause  is  very  unlikely  to  turn  away;  instead  the  response  will  more  likely 
be  a.  panic  stricken  rush  towards  a  crowded  area  in  a  public  space  or  at  least  an  area  maimed  by  security  or 


defense  personnel  at  a  border  checkpoint  or  barrier.  In  such  eases,  additional  sensors,  such  as  metal  detectors  and 
infrared  scanners  may  be  deployed  to  confirm  the  threat  and  suitable  preventive  measures  taken  to  effectively 
neutralize  the  threat  or  at  least  minimize  the  destruction  to  life  and  property. 

Security  personnel  are  also  interested  in  apprehending  people  attempting  to  smuggle  contraband  and  pro- 
liibited  items  into  the  country  at  international  airports,  seaports,  and  borders.  Alerting  stimuli  triggered  by 
suspicious  behavior  may  prompt  a  smuggler  at  an  airport  to  attempt  to  hide  or  leave  the  passenger  concourse 
altogether  for  the  aircraft  parking  areas  and  taxiways  through  the  secured  entry  points  and  ramps.  The  miscreant 
can  be  easily  apprehended  in  such  cases.  At  seaports  and  at  borders,  the  decision  of  a  suspect  to  alter  course  and 
try  an  alternate  route  to  evade  authorities  is  a  reliable  indicator  of  intention.  At  land  borders,  suspects  could 
turn  bade  to  the  country  of  origin  but  still  be  apprehended  by  the  authorities  on  their  own  territory. 

Now  consider  the  effect  of  an  alert  stimulus  on  a  false  positive,  a  regular*  benign  human  subject.  Radical 
changes  in  behavior  such  as  trying  to  escape  or  evade  authorities  or  breaking  into  a  frenzied  rush  axe  typically 
uncommon  responses.  In  most  cases,  such  subjects  may  stop  or  adopt  similar  cooperative  behavior  patterns. 
Passengers  at  an  airport  msliing  to  catch  a  flight  or  hassled  by  ddays  and  long  or  inconvenient  travel  sdiedulcs 
arc  more  likely  to  adopt  a  hostile  or  defensive  attitude  in  response,  but  again  they  arc  very  unlikely  to  adopt  the 
behavioral  patterns  of  terrorists  or  felons.  At  a  military  checkpoint  or  barrier,  alerting  stimuli  may  inconvenience 
or  irritate  a  falsely  identified  subject  but  again,  will  not  inconvenience  the  people  at  large. 

Note  how  alerting  stimuli  in  these  examples  can  positively  identify  a  human  subject  with  malicious  or  hostile 
intent  from  among  a  crowd  even  before  they  approach  the  security  checkpoints.  Valuable  moments  of  time, 
which  may  mean  the  difference  between  destruction  and  prevention,  may  be  gained  by  this  approach.  Inconve¬ 
nience  to  the  public  at  large  will  be  minimized;  sometimes  they  may  even  cooperate  in  the  apprehension  of  a 
suspect  on  the  move  before  the  arrival  of  security  personnel.  It  should  be  noted  because  of  cost  considerations, 
sophisticated  sensors  may  not  be  widely  deployed  in  all  public  spaces  except  at  places  of  vital  importance  such 
as  airports,  government  buildings,  and  famous  monuments  and  landmarks.  In  such  cases,  alert  stimuli  generated 
by  inexpensive  derices  alone  can  be  very  effective  in  identifying  potential  threats  to  public  security. 

Across  international  borders  (e.g.,  U.S.-Canada  and  U.S. -Mexico)  where  much  of  the  length  of  the  border 
is  strewn  ("dusted")  with  vibration  sensors  or  profiling  sensors,  strategically  positioned  derices  that  generate 
alert  stimuli  may  prompt  individuals  to  alter  their  course  and  redirect  them  to  selective  locations  (Section  2.3) 
wfhere  more  capable  and  well-camouflaged  sensors,  such  as  metal  detectors  and  infrared  cameras  are  placed. 
Whether  the  subjects  have  benign  or  malignant  intentions,  such  subjects  can  be  better  identified  and  intercepted 
as  necessary.  Course  altering  behavior  can  also  help  in  determining  the  intent  of  groups  of  people  as  discussed 
in  the  next  subsection. 

2.2  Alert  Stimuli  and  Responses  in  Groups  of  People 

Groups  of  people  driven  by  hostile  intent  at  a  militarized  zone  may  be  harder  to  identify  if  they  use  very  effective 
concealment  techniques.  Consider  a  protesting  group  of  people  just  outside  a  military  base  or  at  a  checkpoint 
that  includes  a  well-concealed  gunman  or  someone  brandishing  a  Molotov  cocktail.  Such  threats  are  veiy  hard 
to  identify  by  conventional  surveillance  techniques.  Alerting  stimuli  may  not  be  of  much  use  in  such  situations, 
except  in  case  someone  panics. 

At  international  borders,  a  variety  of  stimuli  may  be  used  to  identify  telltale  behavioral  patterns.  A  simple 
audible  alert  or  flashing  lights  may  cause  harmless  subjects,  villagers  walking  along  the  border  on  their  way  home 
for  example,  to  call  out  to  identify  themselves  or  stop  or  similar'  cooperative  behavior.  Subjects  attempting  an 
illegal  crossing  may  attempt  to  hide,  change  course,  or  damage  and  destroy  any  equipment  that  they  can  identify. 
Occasionally,  they  may  even  go  back  the  way  they  came.  Diversions  from  the  course  may  have  the  desired  effect 
of  concentrating  a  large  number  of  people  into  a  small  area  (Section  2.3)  where  more  advanced  sensing  techniques 
may  be  deployed  to  determine  if  they  are  wearing  uniforms,  armed,  or  canning  contraband.  Deployed  advanced 
sensors  may  be  activated  opportunistically  in  such  cases  to  refine  the  situation  assessment. 

Li  a  border  surveillance  context,  additional  stimuli  could  be  used  apart  from  the  auditory  and  visual  stimuli. 
Trip  wares  could  be  raised  to  cause  some  members  of  a  group  to  fall.  Tliis  may  cause  a  variety  of  soimds  to  be 
detected  by  a  sensitive  microphone,  such  as  metallic  sounds  if  the  group  is  carrying  guns.  An  arm  in  a  vertical 


position  could  be  caused  to  drop.  Such  events  can  cause  predictable  responses  in  the  group  from  which  intent 
may  be  putatively  assessed.  An  armed  group  advancing  towards  a  checkpoint  with  the  intention  to  attack  may 
freeze,  scan  the  area,  and  wait  for  other  sounds.  Or  they  may  decide  to  press  on  full  speed  ahead  to  try  and 
storm  the  outpost,  hi  either  case,  more  sensors  may  be  activated  to  study  the  group.  If  additional  sensed 
data,  confirms  the  armed  status  of  the  group,  more  aggressive  countermeasures  can  be  employed.  The  choice  of 
stimulus  may  also  decide  the  type  of  response  from  the  group.  The  sound  of  a  rifle  being  chambered  may  elicit 
a  more  aggressive  response  from  an  armed  group  than  the  sound  of  a  breaking  twig  or  a  birdcall;  for  example, 
freeze  and  scan  in  the  latter  case  as  opposed  to  war  cries  and  charge  in  the  former. 

2.3  Constrained  Environments  for  Border  Surveillance 

Simple  devices  that  alert  human  subjects  can  be  used  in  a  landscaped  environment,  effectively  diverting  human 
subjects  into  a  small  area  where  state-of-the-art  sensing  devices  have  been  deployed.  For  example,  consider  the 
harsh  desert  conditions  typical  of  much  of  the  U.S. -Mexico  border.  Possible  route  choices  for  human  subjects 
at  the  border  can  be  artificially  constrained  by  a  synthesized  lake  for  example,  or  by  the  cultivation  or  trans¬ 
plantation  of  thorny  desert  plants,  such  as  cacti  that  restrict  movement  through  the  plants.  At  night,  effective 
use  of  lighting  can  be  used  to  similar  effect.  The  area  can  be  landscaped  with  lakes,  thorny  plants,  and  other 
artifacts  such  that  human  subjects  are  forced  to  move  towards  constrained  areas  where  more  advanced  sensing 
devices  have  been  deployed,  hi  the  example  of  the  artificial  lake,  thirsty  human  subjects  in  a  desert  environment 
are  very  likely  to  be  drawn  to  the  water.  Access  to  the  water  itself  can  be  restricted  to  one  (or  a  few  more) 
points  where  advanced  sensors  are  deployed.  These  ideas  can  leverage  existing  work  in  the  use  of  landscaping  for 
crime  prevention  and  surveillance  from  a  criminal  justice  perspective.11  Such  ideas  will  also  require  collaborative 
work  with  researchers  in  behaiioral  and  cognitive  science,  as  well  as  with  experts  in  ecology  and  environmental 
science. 


3.  FORMALIZING  CONTEXTS,  ALERT  STIMULI,  RESPONSES,  AND  INTENTS 

The  previous  section  discussed  the  possible  contexts  in  which  alerting  stimuli  could  be  applied  to  determine  the 
intent  of  human  subjects,  the  variety  of  stimuli  that  could  be  used,  the  possible  responses  to  these  stimuli,  and 
how  these  could  be  used  as  indicators  of  intent.  The  purpose  of  the  next  section  is  to  begin  the  presentation  of  a 
formal  approach  to  defining  these  contexts,  stimuli,  responses,  and  intents  towards  inclusion  of  these  definitions 
in  a  Semantic  Web  ontology.12 

3.1  An  Ontology  for  Response-based  Intent  Detection 

Semantic  Web  ontologies  contain  formal,  logic-based  definitions  of  concepts  and  inter-concept  relationships  that 
are  pertinent  to  a  knowledge  domain.  Each  of  the  defined  concepts  and  relationships  is  assigned  a  specific  Uniform 
Resource  Identifier  (URI),  which- is  an  accessible  location  on  the  Web.  The  ontology  containing  the  definitions 
has  a  unique  Web  address  and  each  of  the  defined  concepts  and  relations  corresponds  to  a  specific  subsection  of 
this  address.  An  ontology  for  intent  detection  will  include  definitions  of  contexts,  stimuli,  behavioral  responses, 
and  intents.  Each  of  these  concepts  is  discussed  in  the  following  subsections. 

3.1.1  Contexts 

Response-based  intent  detection  can  be  used  in  many  contexts.  Contexts  are  defined  as  combinations  of  locations 
and  times.  Times  are  simple  attributes  that  can  take  on  a  value  from  a  set  of  values  including  wartime  and 
peacetime.  Locations  include  international  land  and  sea  borders,  airports,  military  bases,  seaports,  government 
buildings,  national  monuments,  famous  landmarks,  and  other  areas  of  vital  importance.  Figure  2  shows  a  prelim¬ 
inary  definition  of  the  Application^  Context  concept  using  constructs  from  the  Web  Ontology  Language  (OWL). 
Note  OWL  concepts  are  the  same  as  UML  classes.  These  words,  concepts  and  classes,  are  used  interchangeably 
in  this  paper. 


<owl :  Class  rdf :  about=M#  Application  ..Context  "> 

<rdfs:subClassOf> 

<ovl : intersectionOf  rdf :parseType=MCollectionu> 

<ovl :  Restriction 

<ovl :  onProperty  rdf : resour  ce=,,#inTiine"/> 

<owl :  cardinality  rdf : datatype=,,&xsd;NonNegativeInteger,,>  1  </owl:cardinality> 
</owl :  Restriction 
cowl : Restriction> 

Cowl .* onProperty  rdf : resource=n#atLocation"/> 

Cowl :c  cardinality  rdf : datatype="&xsd;NonNegativeInteger">  1  C/owl  rear din ality> 
</ owl : Restriction> 
c/owl : intersectionOf > 
c/rdf s : subClassOf > 
c/owl ;Class> 


Figure  2.  OWL  definition  for  Application _  Context 


3.1.2  Alert  stimuli 

Alert  stimuli  are  broadly  categorized  into  auditory,  visual,  and  others.  From  the  discussion  in  Section  2,  eaeh  of 
these  categories  may  be  defined  as  sub-concepts  of  the  Stimulus  concept  or  they  may  be  defined  as  enumerated 
values.  Given  that  several  types  of  auditor}'  and  visual  stimuli  can  be  used,  it  makes  sense  to  define  them  as 
full-fledged  sub-conccpts  of  the  Stimulus  concept  as  shown  in  Figure  3.  The  sub-concepts  can  be  instantiated  by 
stimuli,  such  as  flashing  lights  (Visual^ Stimulus  instance)  and  a  trip  wire  being  raised  (Others  instance). 


Figure  3.  Alert ^  stimulus  class  hierarchy. 


3.1.3  Responses 

Possible  responses  of  human  targets  to  alert  stimuli  include  freezing,  calling  out,  charging,  wax  cries,  panicking, 
changing  course  of  movement,  hiding,  rushing  forward,  going  back  the  way  they  came,  and  attempting  to 
sabotage  or  destroy  surveillance  equipment.  These  actions  can  all  be  instances  of  the  Response  concept  in  an 
OWL  ontology. 

3.1.4  Intents 

Intents  can  be  classified  as  hostile  or  non-host ile  at  a  very  high  level.  Assessment  of  intent  is  dependent  upon  the 
alert  stimulus  that  was  incident  upon  the  human  subject (s),  the  context  (the  location  and  time)  when  it  was  used, 
and  the  recorded  behavioral  responsc(s)  of  the  subjcct(s)  to  the  stimulus.  This  can  be  defined  as  a  production 
rule  where  each  of  the  parameters  that  influence  the  assessed  intent  is  an  antecedent  while  the  assessed  intent  is 
the  consequent.  Note  the  intent  can  also  be  defined  as  a  restriction  class  (named  Assess ed_ Intent)  hi  OWTL  with 
three  essential  associations  with  the  Application _  Context,  Alert _ Stimulus,  and  Response  concepts  as  shown  in 
Figure  4.  Figure  5  shows  an  example  of  a  production  rule  conforming  to  these  associations  shown  in  Figure  4. 


Hostile  Intent 


N  on-Hostile  Intent 


Assessed  Jntent 


Figure  4.  Assessed^  intent  class. 


instanceOf (? intent ,  Assessed_Intent) ,  responseFor Intent (? charge,? intent) , 
st imulusF or Intent (?sound,? intent) ,  contextFor!ntent(?appCtx,  ?intent) , 
instanceOf  (?appCtx,  Application_Context) ,  inTime(?appCtx,  ''War"), 
atLocation(?appCtx,  ’’Border’1),  instanceOf  (?sound,  Audi  tory_  Stimulus)  , 
audit or yStimulusType(?sound, "rif le.loaded") ,  instanceOf (?charge,  Response) , 
responseType(? charge,  "charge") 

instanceOf (?intent,  Hostile Intent) . 


Figure  5.  Production  rule  for  inferring  Assessed^  Intent 

Note  each  of  the  attributes  in  the  classes  Response  and  Alert__  Stimulus  can  be  defined  as  OWL  datatj^pe 
properties,  which  take  enumerated  string  values.  However,  ea-ch  of  these  values  can  be  reified  into  classes  of  their 
own  in  the  long  term  following  best  practices  recommendations. 

3*2  Automation  of  Intent  Detection 

The  development  of  an  ontology,  with  concepts  described  in  the  last  section,  is  central  to  the  development  of 
a  daia-to-decision  framework  to  automatically  provide  an  assessment  of  the  threat  posed  by  human  subject(s) 
under  surveillance.  The  framework  would  likely  use  production  rules  implemented  in  Jess13  or  other  similar 
rule-based  engines  to  provide  a  numerical  estimate  of  the  likelihood  of  threat  from  the  human  subject (s)  under 
surveillance.  In  the  interest  of  reusability  and  interoperability,  we  propose  the  use  of  Semantic  Web14  compatible 
probabilistic  reasoning  engines,  such  as  Pronto,15  an  implementation  of  Semantic  Web  compatible  uncertainty 
reasoning*.16 


4.  RELATED  WORK  AND  CONCLUSIONS 


The  idea  of  using  human  behavior  to  identify  hostile  intent  and  deception  is  a  very  old  one.  A  system  capable 
of  detecting  intent  from  video  surveillance  footage  was  patented  in  1997.17  Cohen  et  a l.4  have  proposed  an 
intent  detection  system  using  Point-light  Walker  (PLW)  displays  of  human  motion.  This  work  leverages  previous 
research  into  identifying  the  emotional  state  of  humans  using  motion  cues.18  Tracked  human  movement  is 
compared  with  recorded  data  from  actors  in  a  simulated  environment  as  proposed  by  Rank  et  al.19 

Alerting  stimuli  may  significantly  impact  the  efforts  to  remotely  assess  the  intent  of  human  subjects.  In  the 
examples  discussed  in  this  paper  (Section  2),  alert  stimuli  of  different  kinds  are  capable  of  eliciting  predictable 
responses  from  individuals  and  groups  under  surveillance,  all  of  which  may  not  be  possible  with  conventional 
sensing  methodologies.  They  have  the  desirable  effect  of  necessitating  the  use  of  more  advanced  and  power 
consuming  sensing  tedmiques  only  when  required  and  only  to  refine  the  assessment  of  threat.  In  addition,  alert 
stimuli  can  potentially  evoke  telltale  responses  from  suspects  well  in  advance  of  their  approaching  a  sensitive 
checkpoint  or  bander,  improving  the  prospects  of  effective  countermeasures  and  threat  neutralization.  Inconve¬ 
niences  caused  to  the  general  public  at  large  by  the  current  practice  of  security  checkpoint  profiling  will  also  be 
significantly  reduced. 

A  data-to-decision  framework  for  estimating  the  likelihood  of  hostile  intent  in  human  subject (s)  based  upon 
the  responses  of  the  subjects  to  an  alerting  stimulus  has  been  outlined.  A  number  of  application  contexts, 
alerting  stimuli,  and  possible  responses  to  the  stimuli  have  been  discussed  and  defined  in  a  formal  Semantic 
Web  ontology.  This  ontology  is  necessary  to  automate  the  process  of  tlrreat  identification  and  assessment- in  tire 
future. 
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Abstract.  In  the  absence  of  detector  arrays,  a  single  pixel  coupled  with  an 
image  plane  coded  aperture  has  been  shown  to  be  a  practical  solution  to 
imaging  problems  in  the  terahertz  and  sub-millimeter  wave  domains.  The 
authors  demonstrate  two  laboratory,  real-time,  two-dimensional,  sub¬ 
millimeter  wave  imagers  that  are  based  on  an  image  plane  coded  aper¬ 
ture.  These  active  imaging  systems  consist  of  a  heterodyne  source  and 
receiver  pair,  image  forming  optics,  a  coded  aperture,  data  acquisition 
hardware,  and  image  reconstruction  software.  In  one  of  the  configurations, 
the  target  is  measured  in  transmission,  while  in  the  other  it  is  measured  in 
reflection.  In  both  configurations,  images  of  the  targets  are  formed  on  the 
coded  aperture,  and  linear  measurements  of  the  image  are  acquired  as 
the  aperture  patterns  change.  Once  a  sufficient  number  of  linearly  inde¬ 
pendent  measurements  are  obtained,  the  image  is  reconstructed  by  sol¬ 
ving  a  system  of  linear  equations  that  is  generated  from  the  aperture 
patterns  and  the  corresponding  measurements.  The  authors  show  that 
for  image  sizes  envisioned  for  many  current  applications,  this  image 
reconstruction  technique  is  computationally  efficient  and  can  be  imple¬ 
mented  in  real  time.  Measurements  are  collected  with  these  systems, 
and  the  reconstruction  results  are  presented  and  discussed.  02012  Society 
of  Photo-Optical  Instrumentation  Engineers  (SPlE),  [DOl:  10.1117/1.OE.51  9.091612] 
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1  Introduction 

The  interest  in  terahertz  imaging  is  motivated  by  the  ability 
of  terahertz  (THz)  frequencies  to  penetrate  most  manmade 
materials,  particularly  clothing.1-3  Additionally,  many  harm¬ 
ful  chemicals  and  explosives  display  absorption  lines  that 
make  them  identifiable  by  THz.  ,5  These  characteristics  of 
THz  make  it  attractive  for  security  applications.  Unfortu¬ 
nately,  THz  imaging  is  one  of  the  last  electromagnetic  ima¬ 
ging  modalities  for  which  focal  plane  arrays  are  still 
impractical  and/or  too  costly  to  be  used  in  practical  security 
applications.  An  intermediate  solution  has  been  the  develop¬ 
ment  of  scanning  systems.  These,  however,  have  their  own 
cost,  weight,  power,  and  frame  rate  limitations. 

Our  interest  is  focused  in  the  part  of  the  THz  spectrum 
between  300  GHz  and  3  THz,  which  is  referred  as  the 
sub-millimeter  wave  region.  This  interest  is  motivated  by 
the  availability  of  mature  single  detector  technologies  at 
these  frequencies.  Both  active  and  passive  imaging  config¬ 
urations  at  these  frequencies  have  been  successfully  demon¬ 
strated,  and  now  the  new  frontier  in  sub-millimeter  wave 
imaging  is  faster  image  frame  rate  and  smaller  size,  weight, 
power  consumption,  and  cost.  Passive  systems  that  operate  at 
close  to  real  time  require  cooled  detectors  and  tend  to  have 
very  large  form  factors.6,7  Images  from  these  systems  are 
similar  to  infrared  images  and  provide  temperature  differ¬ 
ence/contrast  information.  On  the  other  hand,  the  images 
acquired  with  active  imagers,  that  are  usually  coherent, 
have  very  high  dynamic  range  and  suffer  from  the  specular 
reflections  and  clutter  in  the  scene.  However,  this  phenom¬ 
enology  can  be  mitigated  by  using  the  imager  in  radar  mode 
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as  demonstrated  by  Cooper.8  An  advantage  of  active  systems 
is  that  they  operate  at  room  temperature  and  their  form  factor 
is  usually  limited  by  the  optical  aperture  rather  than  the 
detector  and  source  modules.  This  makes  active  systems 
attractive  for  applications  that  require  the  imager  to  be 
portable  and  operate  at  stand  off  ranges. 

Awaiting  the  development  of  practical  and  inexpensive 
focal  plane  arrays  for  sub- millimeter  wave  imaging,  other 
alternatives  that  exploit  the  maturity  of  the  single  pixel  detec¬ 
tor  have  been  explored.  In  particular,  we  have  focused  our 
efforts  on  spatial  light  modulators  (SLM)  for  sub-millimeter 
waves  and  their  application  as  image  plane  coded  apertures.9,10 
Spatial  light  modulators  work  in  reflection  mode  or  in  trans¬ 
mission  mode.  Perhaps  the  most  well-known  reflective  SLM  is 
the  Texas  Instruments  digital  micro-mirror  device  (DMD) 
known  for  its  use  in  the  Rice  University  single  pixel  camera,1 1 
or  in  most  modem  day  projectors.  A  reflective  SLM  for 
sub-millimeter  waves  has  been  proposed  by  Lukkanen12 
and  entails  using  an  electronically  controlled  reflect-array 
that  can  be  used  as  a  beam  steerer.  However,  this  work  is  in 
its  design  phase. 

An  example  of  transmissive  SLM  is  the  liquid  crystal 
optical  modulators  common  in  display  applications.  There 
have  been  some  efforts  to  use  metamaterials  to  develop  an 
electronically  controlled  transmissive  SLM13  for  THz,  but 
the  modulation  depth  of  that  device  did  not  exceed  40%, 
with  a  minimum  transmission  coefficient  of  0.3  and  maxi¬ 
mum  of  0.7.  Higher  modulation  depths  are  preferred  for 
most  sub-millimeter  wave  imaging  applications.  Another 
approach  that  is  being  investigated  for  transmissive  THz 
modulators  is  the  use  of  microelectromechanical  systems 
(MEMS)  reconfigurable  sub-wavelength  metallic  slits. 
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Although  this  work  is  promising,  but  so  far  only  analysis 
and  small  scale  devices  for  proof  of  concept  have  been 
demonstrated.1 4,15 

The  lack  of  high  power  illumination  sources  in  the  sub- 
millimeter  wave  regime  and  the  atmospheric  absorption 
require  special  consideration  for  the  efficiency  of  SLMs 
for  sub-millimeter  waves.  Practical  implementations  of 
optical  systems  for  SLMs  where  the  image  plane  is  being 
modulated  favor  transmission  mode  devices.  The  energy 
that  passes  through  the  SLM  transmission  mode  device 
can  be  collected  and  channeled  to  the  detector,  whereas 
the  energy  reflected  by  the  SLM  reflection  mode  device 
will  be  diffracted  away  from  the  detector. 

We  have  developed  a  mechanically  scanned  SLM  for  sub¬ 
millimeter  waves  that  serves  as  an  image  plane  coded  aperture. 
Although  a  mechanically  scanned  system,  our  device  has  low 
mechanical  complexity,  provides  high  modulation  levels  and 
high  scanning  rates,  and  has  low  cost  At  the  core  of  this 
device  is  a  rotating  disk  with  holes  that  spins  at  a  constant 
speed.  The  image  is  formed  on  one  portion  of  the  disk  and 
is  transmitted  through  the  holes  onto  the  single  pixel  receiver. 
Linear  measurements  are  made  on  the  image  as  the  disk 
rotates.  An  image  is  reconstructed  using  the  knowledge  of 
the  hole  patterns  and  the  measured  signal  from  the  receiver. 

We  have  previously  presented  the  electromagnetic  analy¬ 
sis  and  design  of  this  device  as  well  as  a  working  example  of 
a  line  image  plane  coded  aperture  as  part  of  a  line  imager.9,10 
In  this  paper  we  describe  and  demonstrate  a  two-dimensional 
version  of  the  device  that  is  used  to  form  64  x  64  pixel 
sub- millimeter  wave  images.  Sub-milli meter  wave  images 
are  formed  in  transmission  mode  and  in  stand-off  reflection 
mode.  In  transmission  mode,  a  transmissive  target  is  pro¬ 
jected  onto  the  device.  In  stand-off  mode,  the  target  is  placed 
10  m  away  from  the  imager  and  an  image  of  it  is  formed  on 
the  device  through  an  optical  system.  The  next  section  intro¬ 
duces  the  image  plane  coded  aperture  device  and  the  math¬ 
ematical  formulation  of  the  forward  imaging  problem  with 
this  coded  aperture.  In  Sec.  3,  we  present  an  image  recon¬ 
struction  method  and  discuss  the  effect  of  the  reconstruction 
parameters  on  the  relative  reconstruction  error  of  the  solu¬ 
tion.  We  present  results  from  simulations  for  two  imaging 
configurations  of  interest.  In  Sec.  4  we  describe  the  imple¬ 
mentation  of  the  image  plane  coded  aperture  and  the  imaging 
setups.  Images  formed  using  the  device  are  presented  and 
discussed  in  Sec.  5.  Based  on  the  discussion  in  Sec.  5 
and  the  results  from  the  simulations  presented  in  Sec.  3, 
we  conclude  with  some  recommendations  for  future  work. 

2  Two-Dimensional  Image  Plane  Coded  Aperture 

A  conceptual  sketch  of  the  image  plane  coded  aperture  is 
shown  in  Fig.  1.  The  device  is  placed  in  the  image  plane 
of  the  imager,  and  an  image  is  formed  over  the  imaging  win¬ 
dow.  The  image  is  scanned  with  a  spinning  disk  that  is  pat¬ 
terned  with  holes.  The  detector  is  sampled  continuously  and 
each  measurement  sample  is  associated  with  the  pattern  of 
holes  in  the  imaging  window  at  the  sample  instant.  Once 
the  required  measurements  are  recorded,  the  image  is  recon¬ 
structed.  The  central  component  of  the  coded  aperture  is  the 
spinning  disk  with  holes  located  at  random  positions.  The 
placement  of  the  holes  and  their  sizes  depend  on  the  largest 
imaging  wavelength,  X^.  Electromagnetic  analysis10  has 
shown  that  for  a  disk  of  thickness  around  three  Ximx  the 


Fig.  1  Conceptual  drawing  of  the  coded  aperture  placed  in  the  image 
plane  of  the  imaging  system.  Insert:  arbitrary  division  of  the  imaging 
window  into  pixels. 

minimal  hole  radius  should  be  greater  than  lAX^  and 
the  minimal  edge  to  edge  separation  between  holes  should 
be  one  X^.  As  long  as  the  radius  of  the  holes  and  the 
depth  increase  proportionally,  transmission  and  linearity 
are  preserved.  Because  of  the  assumptions  of  the  image 
reconstruction  method,  the  transmission  of  each  hole  must 
be  unity,  and  transmission  from  many  holes  must  be  linear. 

Another  component  of  importance  is  the  collecting  horn 
placed  behind  the  imaging  window  and  the  disk.  This  col¬ 
lecting  mechanism  is  necessary  for  coupling  all  of  the  energy 
that  makes  its  way  through  the  holes  into  the  detector.  An 
integrating  sphere  can  also  be  used  when  the  signal  is  abun¬ 
dant,  but  this  is  rarely  the  case  with  active  sub-millimeter 
wave  systems.  Alternatively,  if  a  horn  or  integrating  sphere 
is  not  available,  the  measured  signal  should  be  a  good  repre- 
sentatio'n  of  the  energy  through  the  holes.  By  measuring  the 
transmitted  field  in  the  far  field  of  the  imaging  window  or  by 
use  of  a  lens,  the  received  signal  becomes  representative  of 
the  total  energy  in  the  signal.  This  effect  is  a  direct  result  of 
the  forward  scattering  theorem. lb 

The  imaging  window  is  another  component.  It  is  an  aper¬ 
ture  that  limits  the  part  of  the  disk  where  the  image  is  formed 
and  scanned.  The  size  of  the  imaging  window  determines  the 
physical  size  of  the  image.  It  can  be  of  any  shape;  two  shapes 
that  are  considered  in  this  paper  are  square  and  narrow 
rectangle  (slit). 

2.1  Mathematical  Formulation  of  the  Imaging 
Problem 

For  reconstruction  purposes,  the  imaging  problem  is  posed 
as  a  linear  system  of  equations  and  is  presented  in  matrix 
notation 

Mp  +  rj  =  m  =  m  +  rj.  (1) 

For  an  image  of  n  X  n  pixels,  the  column  vectors  m,  m,  rj , 
have  dimensions  k  X  1,  the  column  vector  p  has  dimensions 
n2  X  1,  and  the  matrix  M  is  a  matrix  of  dimensions  kx  n2. 
The  vector  p  is  composed  of  the  values  of  each  image  pixel. 
The  vector  rh  =  m  -f  rj  is  composed  of  the  values  of  each 
measurement  and  is  called  the  measurement  vector.  The  vec¬ 
tor  m  is  the  noiseless  measurement  and  77  is  the  measurement 
noise.  The  number  of  measurements  is  k.  The  matrix  M  is 
called  the  measurement  matrix  and  each  of  its  rows  encodes 
the  pattern  of  holes  over  the  imaging  window  for  each  mea¬ 
surement.  In  this  work  we  consider  the  case  when  k  =  n1  and 
the  measurement  matrix  is  square.  Referring  to  the  insert  in 
Fig.  1,  each  entry  of  each  row  of  the  measurement  matrix  is 
calculated  as  the  fraction  of  the  area  of  the  corresponding 
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pixel  covered  by  holes.  For  instance,  the  measurement  matrix 
entries  M\t\  =  0.23,  Af12  =  0  27,  M1(3  =  0,  correspond  to 
image  pixel  entries  plt  p2,  and  /?3  in  the  insert  and  represent 
how  much  of  the  area  of  each  pixel  is  covered  by  the  holes. 
As  the  disk  rotates  and  a  new  measurement  is  acquired,  the 
measurement . matrix  entries  change  to  reflect  the  area  of 
intersection  between  pixels  and  holes.  Linear  equations 
can  be  formed  using  these  measurement  matrix  entry  values, 
the  unknown  pixel  values,  and  the  measurement  values  or 

M nPl  +  •  •  •  +  M\nPn  +  •  •  •  +  M 1  ,,2/V  = 

...  =m...  (2) 

Mn*\P\  +  . . .  +  Mntnpn  +  . . .  +  =  'V* 


Interpreting  these  linear  equations,  the  entries  of  the  mea¬ 
surement  matrix  can  also  be  understood  as  the  contribution 
of  each  pixel  to  the  measurement.  The  sizes  of  the  pixels  are 
arbitrary;  therefore,  the  imaging  window  can  be  divided  into 
as  many  pixels  as  desired.  However,  as  the  number  of  pixels 
increases,  the  rows  of  the  measurement  matrix  become  less 
linearly  independent.  The  linear  independence  of  the  equa¬ 
tions  whose  coefficients  are  represented  in  the  rows  of  the 
measurement  matrix  is  assumed  in  order  to  solve  Eq.  (1). 
The  pixels  can  also  take  any  shape  and  are  not  limited  to 
the  square  shape  shown  in  the  insert  of  Fig.  1. 

Ideally,  the  mathematical  formulation  of  the  imaging 
problem  requires  that  the  measurement  samples  be  associated 
with  the  pattern  of  holes  (the  measurement  matrix  row)  in  the 
imaging  window  at  the  sample  instant  In  the  actual  implemen¬ 
tation  the  precise  location  of  the  holes  is  only  known  to  within  a 
constant  offset  angle.  Equations  (1)  and  (2)  can  be  interpreted  as 
a  m  atrix  representation  of  the  correlation  of  the  disk  hole  pattern 
with  the  image  discretized  in  pixels.  By  empirically  testing 
assumed  offset  angles  in  the  formulation  of  the  measurement 
matrix  and  checking  the  resulting  image  registration  for  a  cali¬ 
bration  image  (a  point  source),  this  unknown  can  be  found. 

In  constructing  the  measurement  matrix  in  Eq.  (1)  we 
make  the  following  assumptions.  Each  measurement  is 
collected  as  the  pattern  of  holes  in  the  imaging  window 
is  stationary.  All  the  energy  that  passes  through  the  holes 
is  collected  and  measured  by  the  detector.  The  measured 
energy  is  the  sum  of  the  energy  passing  through  all  the 
holes.  However,  all  of  these  assumptions  are  violated  by 
the  actual  implementation.  The  disk  is  rotating  at  a  constant 
speed,  and  each  measurement  is  collected  as  the  disk  rotates. 
The  majority,  but  not  all,  of  the  energy  that  goes  through  the 
holes  is  collected  and  measured.  However,  in  the  current 
implementation  the  error  introduced  by  the  approximate 
measurement  model  is  overshadowed  due  to  the  measure¬ 
ment  noise.  Both  the  noise  and  error  due  to  the  measurement 
model  approximation  are  mitigated  by  the  regularization  that 
is  introduced  in  solving  Eq.  (1). 

3  Image  Reconstruction 

The  unknown  pixel  values  are  obtained  by  solving  Eq.  (1).  In 
the  presence  of  noise  and  inaccuracies  in  the  coefficients  of 
the  measurements  matrix  only  an  approximate  solution,  p, 
can  be  obtained  because  the  problem  is  ill-conditioned.  A 
number  of  solution  methods  can  be  borrowed  from  the 
inverse  imaging  literature  as  well  as  from  the  compressive 


sensing  literature.17  Most  methods  however,  are  iterative 
in  nature  and  therefore  not  suited  for  real-time  systems. 
For  real-time  systems,  closed  form  solutions  are  preferred. 
One  method  for  finding  the  approximate  closed  form  solu¬ 
tion  is  the  regularized  least  squares18 


P  = 


MrM  + 


-1 


Mrm  =  R  {pi)m% 


(3) 


where  ps  are  regularization  parameters,  H,-  are  linear  opera¬ 
tors  applied  on  the  solution,  and  the  superscript  T  denotes  the 
matrix  transpose.  Borrowing  notation  from  Bertero  and  Boc- 
cacci,18  the  resulting  matrix  multiplication  and  additions  axe 
all  grouped  together  into  R(//,-)  for  notational  ease.  In  the 
absence  of  any  regularization  R(/^  =  0)  =  M-1,  is  the 
inverse  of  M.  The  linear  operators  H,  are  modeled  using 
a  priori  knowledge  of  the  solution.  For  example,  if  the  mini¬ 
mal  norm  solution  is  desired,  the  linear  operator  is  the 
identity  matrix.  If  the  solution  is  smooth,  the  linear  operator 
could  be  the  Laplacian  so  that  sharp  transitions  are 
minimized. 

From  Eqs.  (1)  and  (3)  we  can  write  the  approximate 
solution  in  terms  of  a  noiseless  approximation  term  and  a 
propagated  noise  term 

p  =  R(pi)m  +  R(pth-  (4) 

In  the  absence  of  noise  and  regularization  the  approximate 
solution  reaches  the  exact  solution 

lim  R  (pi)m  +  R  (pArj  =  M~lm  =  p.  (5) 


The  exact  solution  is  defined  by  Eq.  (5)  and  should  be  inter¬ 
preted  as  the  “model  limited”  reconstruction  of  the  image 
plane,  not  the  object  plane.  The  term  “model  limited”  refers 
to  the  fidelity  of  the  measurement  model  represented  by  the 
measurement  matrix  M.  With  the  approximate  and  exact 
reconstructions  defmed,  we  can  express  the  reconstruction 
error  £  as 


e  =  p-p  e  =  [R(/i|)m  -  p]  +  (6) 


The  relative  reconstruction  error  is  a  more  useful  quantity 
and  it  is  given  as 


(7) 


For  the  regularized  least  squares  solution,  the  relative  recon¬ 
struction  error  depends  on  the  measurement  noise,  and  on 
the  regularization  parameters,  pt. 

The  reconstruction  error  of  Eq.  (6)  is  composed  of 
two  terms:  the  approximation  error  [R(^,)m-  p]  and  the 
noise  propagation  [R {ji$)rj\.  Expanding  R (//,•)  as  in 
Eq.  (3),  one  observes  that  the  approximation  error  is  directly 
related  to  the  regularization  parameters,  and  the  noise  propa¬ 
gation  term  is  inversely  related  to  the  regularization  para¬ 
meters.  When  the  measurement  matrix  is  only  mildly  ill- 
conditioned  and  the  regularization  parameters  are  small, 
the  noise  propagation  term  dominates  the  reconstruction 
error.18  Ideally  we  require  measurement  matrices  to  have 
low  condition  numbers  or  to  be  mildly  ill-conditioned. 
Real-time  imaging  is  more  forgiving  of  noise  propagation 
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than  approximation  error.  The  noise  will  appear  to  have  high 
spatial  and  temporal  frequency  spectrum  and  will  be  inte¬ 
grated  out  by  the  eye  of  the  observer  while  the  approximation 
error  will  have  low  spatial  and  temporal  frequency  spectrum 
and  will  appear  as  clutter. 

3.1  Dependence  of  the  Measurement  Matrix  on 
Device  Implementation 

The  analysis  presented  above  indicates  that  low  condition 
number  measurement  matrices  are  desired.  For  the  coded 
aperture  described  here,  the  condition  of  the  measurement 
matrix  is  strongly  related  to  the  size  of  the  disk  and  the  num¬ 
ber  of  measurements  that  are  recorded  for  each  rotation  of  the 
disk.  High  pixel  count  images  require  an  equally  high  num¬ 
ber  of  measurements.  As  the  number  of  measurements 
increases,  the  patterns  used  to  form  the  linear  equations 
represented  in  Eq.  (2)  become  similar  or  correlated,  and 
the  equations  become  less  independent.  This  results  in  an 
ill-conditioned  measurement  matrix.  Three  approaches  can 
be  used  to  mitigate  this  effect  while  keeping  the  resulting 
image  pixel  count  reasonable.  The  first  approach  involves 
changing  the  size  of  the  disk.  The  disk  size  can  be  increased 
while  holding  the  imaging  window  size  constant  and  posi¬ 
tioned  toward  the  perimeter  of  the  disk.  This  will  increase 
the  arc  separation  between  measurements  resulting  in 
more  variation  in  pattern  between  measurements.  Increasing 
the  size  of  the  disk  increases  the  form  factor  and  the  mechan¬ 
ical  and  fabrication  complexity  of  the  device.  Further  analy¬ 
sis  is  required  to  determine  the  limits  of  the  disk  size  that  can 
be  used. 

Another  approach  is  turning  the  imager  into  a  line  imager 
and  coupling  it  with  a  tilting  mirror  to  form  two-dimensional 
images.  For  this  puipose  the  imaging  window  can  be  made 
into  a  vertical  slit  (with  respect  to  the  imaging  window  in 
Fig.  1).  In  this  case  fewer  measurements  arc  acquired  per 
rotation  and  nonsimilar  patterns  are  more  abundant  facilitat¬ 
ing  linear  independence  between  measurements  and  there¬ 
fore  good  reconstruction.  The  tilting  mirror  would  have  a 
small  form  factor  and  would  be  placed  immediately  before 
the  coded  aperture.  This  approach  also  increases  the  mechan¬ 
ical  complexity  of  the  system  but  keeps  the  form  factor  of  the 
system  unchanged.  In  addition,  a  line  imager  is  useful  in  and 
of  itself.  For  example,  it  can  be  used  in  conjunction  with  an 
optical  camera  in  a  hallway  to  scan  personnel  for  concealed 
objects  using  self  motion  of  the  person  to  form  the  THz 
image.  The  system  could  then  superimpose  the  THz  images 
on  the  optical  images  as  personnel  pass  through  the  hallway. 

A  third  approach  is  the  use  of  compressive  sampling  (CS) 
reconstruction  methods.17  In  the  CS  approach  the  number  of 
measurements  in  one  rotation  of  the  disk  can  be  less  than  the 
number  of  pixels  or  k  <  n 2.  This  allows  larger  spacing 
between  measurements  and  hence  less  correlation  between 
measurement  matrix  entries.  Our  preliminary  analyses 
with  the  random  hole  patterns  have  indicated  that  the  result¬ 
ing  measurement  matrices  are  still  correlated  even  when  half 
the  number  of  measurements  are  used  and  they  do  not  satisfy 
the  restricted  isometry  property  (RIP).  However,  combina¬ 
tions  of  different  hole  and  pixel  shapes  might  result  in  mea¬ 
surement  matrices  that  satisfy  the  RIP.  In  addition,  the 
identification  of  a  sparsity  basis  could  allow  the  use  of 
even  fewer  measurements  resulting  in  even  less  measurement 
correlation.  Another  implication  of  using  CS  reconstruction 


methods  is  the  increased  measurement  integration  time  for 
equal  measurement  frame  rates.  This  results  in  higher  mea¬ 
surement  SNR.  Further  study  is  required  to  determine  if  the 
information  gain  in  measurement  decorrelation  and  the 
improvement  in  SNR  outweigh  the  information  loss  because 
of  reduced  measurements.  Also,  another  application-depen¬ 
dent  consideration  is  the  reconstruction  time.  One  disadvan¬ 
tage  of  CS  reconstruction  methods  over  the  closed  form 
methods  is  their  iterative  nature  not  suited  for  real  time 
imaging. 

3.2  Performance  Evaluation  with  Simulations 

To  illustrate  the  effect  of  the  regularization  parameters  on  the 
reconstruction,  we  simulated  the  reconstruction  of  noisy 
measurements  using  Eq.  (3).  We  minimized  both  the 
norm  and  the  Laplacian  of  the  solution.  The  operator  for 
the  norm  is  the  identity  matrix  and  the  operator  for  the  Lapla¬ 
cian  is  the  one  defined  in  Ref.  19.  We  sought  to  reconstruct  a 
32  X  32  pixel  image  of  a  gun  shown  in  Fig.  2(a).  An  image 
size  of  32  x  32  pixels  was  chosen  to  facilitate  fast  calcula¬ 
tion  of  the  relative  reconstruction  error  in  Eq.  (7).  We  simu¬ 
lated  the  reconstruction  using  both  a  two-dimensional 
imaging  window  and  a  line  or  slit  imaging  window.  We 
did  this  to  support  our  comments  about  the  advantage  of 
the  line  imager  over  the  two-dimensional  imager  with  respect 
to  the  quality  of  reconstruction.  The  hole  pattern  we  used  in 
these  simulations  is  the  one  we  implemented,  described  in 
the  next  section  and  shown  in  Fig.  3.  The  imaging  windows 
that  were  simulated  were  a  34  mm  by  34  mm  square  and  a 
34/32  mm  by  34  mm  vertical  slit  (with  respect  to  the  ima¬ 
ging  window  in  Fig.  1).  The  imaging  windows  were  super¬ 
imposed  over  the  hole  pattern  to  form  measurement  matrices 
of  size  1024  X  1024  and  32  X  32,  respectively,  as  the  hole 
pattern  is  rotated. 

For  the  two-dimensional  image  reconstruction,  the  mea¬ 
sured  signal  was  simulated  by  multiplying  the  measure¬ 
ment  matrix  with  the  reshaped  (1024  x  1)  gun  image  values. 
The  measured  signal  was  corrupted  with  zero  mean  white 


«)  (<*) 

Fig.  2  (a)  Target  image,  (b)  two-dimensional  imager  reconstruction 
from  signal  with  20  dB  SNR,  (c)  two-dimensional  imager  reconstruc¬ 
tion  from  signal  with  40  dB  SNR,  and  (d)  line  imager  reconstruction 
from  signal  with  20  dB  SNR. 
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Fig.  3  Implementation  of  the  image  plane  coded  aperture  using  parts 
of  a  5.25  in.  hard  drive. 


Gaussian  noise  (using  the  AWGN  function  in  Matlab)  so  that 
the  resulting  signal  to  noise  ratio  of  the  signal  was  20,  30  and 
40  dB.  The  reconstructed  image  was  obtained  by  multiplying 
the  corrupted  signal  with  the  regularized  matrix  R For 
the  slit  image  reconstruction,  the  measured  signal  for  each 
image  column  was  simulated  by  multiplying  the  measure¬ 
ment  matrix  with  each  column.  Each  of  the  signals  was  cor¬ 
rupted  with  zero  mean  white  Gaussian  noise  so  the  resulting 
signal  to  noise  ratio  of  the  signal  was  20,  30  and  40  dB.  The 
reconstructed  image  was  pieced  together  with  the  reconstruc¬ 
tions  from  each  column  as  the  corrupted  signals  were 
multiplied  with  the  regularized  matrix  R (/q). 

For  both  imager  cases  and  all  the  noise  cases,  the  regu¬ 
larization  parameters  were  varied  iteratively  and  the  relative 
reconstruction  eiror  (RRE)  was  calculated  using  Eq.  (7) 
forming  two-dimensional  plots  of  RRE  versus  the  two 
regularization  parameters.  The  regularization  parameters 
that  yielded  the  minimum  RRE  were  used  to  calculate  the 
reconstructions  shown  in  Fig.  2.  Figure  2  shows  the  target 
image  [Fig.  2(a)]  and  the  reconstructed  images  when  the 
imaging  window  is  square  and  the  SNR  of  the  measured  sig¬ 
nal  is  20  dB  [Fig.  2(b)]  and  40  dB  [Fig.  2(c)],  and  when  it  is  a 
vertical  slit  (SNR  20  dB)  and  the  image  is  scanned  across 
[Fig.  2(d)].  The  plots  of  the  RRE  for  both  parameters  are 
shown  in  Fig.  4.  Figure  4(a)  shows  how  the  RRE  changes 
with  respect  to  the  norm  regularization  parameter  when 
the  Laplacian  regularization  parameter  is  optimal  (the  one 
yielding  minimum  RRE).  Figure  4(b)  shows  how  the  RRE 
changes  with  respect  to  the  Laplacian  regularization  para¬ 
meter  when  the  norm  regularization  parameter  is  optimal. 
Plots  for  the  two-dimensional  and  line  image  reconstruction 
are  shown  for  measurement  signals  with  20,  30  and  40  dB 
SNR.  The  plots  are  shown  in  logarithmic  scale  with  respect 
to  the  regularization  parameters  to  display  the  large  dynamic 
range  of  the  parameters,  which  depends  on  noise  level  and 
imager  configuration. 

For  the  optimal  regularization  parameters,  the  RRE  for  the 
line  imager  is  lower  than  for  the  two-dimensional  imager  for 
all  noise  levels.  For  example,  when  the  SNR  is  20  dB,  the  mini¬ 
mum  RRE  for  the  two-dimensional  imager  reconstruction  is 
0.678  and  for  the  line  imager  reconstruction  is  0. 191,  for  SNR 
30  dB  they  are  0.631  and  0.176,  respectively,  and  for  SNR 
40  dB  they  are  0.527  and  0.105,  respectively.  This  difference 


(a) 


Fig.  4  Relative  reconstruction  error  plotted  versus  the  norm  regular¬ 
ization  parameter  (a)  and  versus  the  Laplacian  regularization  para¬ 
meter  (b)  for  the  two-dimensional  (2-D)  Imager  and  for  the  line 
imager  (1-D)  when  the  measurement  signal  is  corrupted  by  zero 
mean  white  Gaussian  noise.  The  lines  are  slices  of  the  two-dimen¬ 
sional  RRE  plots  when  the  other  parameter  is  optimal  (minimal  RRE). 

is  obvious  in  the  reconstructions  in  Fig.  2(b)  and  2(d).  Also, 
because  the  two-dimensional  imager  problem  is  more  ill-con¬ 
ditioned  than  the  line  imager  problem,  higher  regularization 
values  are  required  to  reconstruct  the  best  two-dimensional 
imager  image  than  the  best  line  imager  image.  As  a  result 
the  reconstruction  error  in  the  two-dimensional  imager  recon¬ 
struction  is  clutter  like  (dominated  by  approximation  error) 
while  the  line  imager  reconstruction  is  noise  like  (dominated 
by  noise  propagation).  This  effect  of  the  optimal  regularization 
parameter  on  the  reconstruction  can  also  be  observed  by  com¬ 
paring  the  two-dimensional  imager  reconstructions  with 
20  dB  SNR  [Fig.  2(b)]  and  40  dB  SNR  [Fig.  2(c)].  As  evident 
from  the  plots  in  Fig.  4,  the  impact  on  regularization  due  to 
the  Laplacian  regularization  parameter  is  greater  than  from 
the  norm  regularization  parameter.  However,  for  values  of  the 
Laplacian  parameter  different  than  the  optimal  the  impact  of 
the  norm  parameter  becomes  more  important. 

As  shown  above,  the  line  imager  performs  better  than  the 
two  dimensional  imager  for  the  particular  hole  pattern  that 
we  have  implemented.  One  assumption  that  was  made  how¬ 
ever,  is  that  the  SNR  for  the  measurement  is  the  same  for  the 
two-dimensional  and  line  configurations.  This  assumption  is 
justified  if  proper  optics  are  used  for  both  imagers  while  the 
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power  of  the  illumination  source  is  held  constant.  A  proper 
optical  system  would  illuminate  and  image  only  the  region  of 
the  object  being  measured  by  the  aperture. 

The  condition  number  of  the  line  imager  can  be  lowered 
even  further  if  a  different  pattern  is  implemented  on  the 
disk.  A  pattern  that  implements  a  simplex  code  mask  (SCM) 
is  possible  with  the  line  configuration.  The  measurement 
matrix  that  results  from  an  SCM  consists  of  Is  and  Os.  Accord¬ 
ing  to  Harwit,20  the  SCM  measurement  matrix  is  the  matrix 
with  the  lowest  condition  number  when  only  entries  of  0 
and  1  can  be  used.  The  condition  of  the  two-dimensional 
imager  can  also  be  lowered  using  a  different  disk  pattern, 
and  such  patterns  will  be  investigated  in  the  future. 

4  Implementation  of  the  Image  Plane  Coded 

Aperture  and  Imaging  Setups 

The  two-dimensional  image  plane  coded  aperture  was  imple¬ 
mented  using  parts  of  a  5.25  in.  hard  drive.  One  of  the  platters 
of  the  hard  dnve  was  used  as  the  spinning  disk.  The  radius  of 
the  holes  and  their  minimal  separation  were  designed  for 
operation  at  640  GHz.  A  hole  radius  of  1  mm  and  a  minimal 
separation  of  2  mm  were  chosen  to  guarantee  unity  transmis¬ 
sion,  linearity,  the  structural  integrity  of  the  disk,  and  because 
of  the  availability  of  standard  tools  in  these  sizes.  With  these 
parameters  as  restrictions,  a  random  pattern  of  431  holes  was 
generated,  and  the  holes  were  drilled  on  a  CNC  mill.  The  case 
of  the  hard  drive  was  cut  and  modified  for  access  to  the  disk  and 
for  mounting  on  the  optical  stages.  The  electronics  of  the 
motor  driver  were  also  modified  so  that  the  disk  could  be 
rotated  at  3000  rotations  per  minute  (RPM).  An  emitter 
diode  and  phototransistor  pair  were  placed  in  the  proximity 
of  the  disk  to  register  the  measurement  samples  with  the  cor¬ 
responding  hole  patterns  in  the  imaging  window.  A  34  mm  X 
34  mm  imaging  window  aperture  was  machined  on  two  slabs 
of  aluminum,  and  they  were  mounted  to  sandwich  the  disk. 
Figure  3  shows  a  picture  of  the  front  and  back  of  the  device 
with  all  the  components  described  above  labeled. 

The  coded  aperture  was  used  in  two  imaging  configura¬ 
tions.  In  the  first  configuration  the  device  was  paired  with  the 
sub-millimeter  wave  receiver  to  form  images  in  transmission 
mode.  A  sketch  and  picture  of  this  setup  are  shown  in  Fig.  5. 
We  use  a  heterodyne  source  and  receiver  pair  from  Virginia 
Diodes.  The  source  and  receiver  operate  at  640  GHz.  In  this 
configuration,  the  640  GHz  source  is  placed  1  m  in  front  of 
the  device  and  is  flood  illuminating  the  imaging  window.  An 
object  mask  is  placed  right  before  the  imaging  window,  and 
the  projected  image  is  scanned  by  the  spinning  disk.  The 
640  GHz  receiver  is  placed  behind  the  disk  and  measures 
the  energy  passing  through  the  holes.  The  intermediate  fre¬ 
quency  (IF)  of  the  receiver  was  down  converted  from  4.8  to 
2.2  GHz  and  this  new  IF  was  supplied  to  a  spectrum  analy¬ 
zer.  The  spectrum  analyzer  was  used  to  band-pass  filter  and 
amplify  the  signal.  A  time  sweep  of  the  signal  triggered  for 
each  rotation  was  generated,  and  the  video  output  of  the 
spectrum  analyzer  was  sampled  at  a  rate  of  250  kHz 
using  a  16  bit  data  acquisition  card  from  Measurement  Com¬ 
puting.  The  disk  rotates  at  50  rotations  per  second  and  5000 
samples  are  recorded  per  rotation.  The  sampled  signal  is 
low-pass  filtered  in  Matlab  and  4096  consecutive  samples 
are  used  to  reconstruct  a  64  x  64  pixel  image  using  Eq.  (3). 

In  the  second  configuration,  the  device  was  paired 
with  the  sub- millimeter  wave  receiver  to  form  images  in 


Fig.  5  Sub-miliimeter  wave  setup  in  transmission  mode,  (a)  sketch 
and  (b)  actual  implementation. 


stand-off  mode.  A  sketch  and  picture  of  this  setup  are 
shown  in  Fig.  6.  The  640  GHz  source  is  located  approxi¬ 
mately  2  m  from  the  target  and  flood  illuminates  it  The 
energy  reflected  by  the  target  is  collected  by  the  imaging  mir¬ 
ror  (in  the  implementation,  the  beam  is  folded  by  two  flat 
mirrors)  10  m  away.  This  mirror  is  a  section  of  an  ellipsoid 
of  rotation  with  one  focus  at  1  m  and  the  other  at  10  m.  The 
image  is  formed  on  the  1  m  side.  The  system  has  an  effective 
diameter  of  0.305  m  (12  in.),  effective  focal  length  of 
0.909  m,  resulting  in  a  F#  of  2.983.  The  magnification  of 


Fig.  6  Sub-miiiimeter  wave  setup  in  stand-off  mode,  (a)  sketch  and 
(b)  actual  implementation. 
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the  system  is  0.1,  the  depth  of  focus  for  coherent  radiation  is 
0.011  m,  and  the  depth  of  field  for  incoherent  radiation  is 
1.016  m.  The  airy  disk  diameter  is  3.75  mm  for  a  wavelength 
of  468.43  /v m,  corresponding  to  the  illumination  frequency 
of  640  GHz.  The  imaging  mirror  forms  an  image  of  the  target 
in  the  imaging  window  of  the  coded  aperture.  The  image  is 
then  scanned  by  the  spinning  disk,  and  the  energy  that  goes 
through  the  holes  is  measured  by  the  640  GHz  receiver.  The 
same  data  collection  setup  as  in  the  first  configuration  is  used 
to  sample  the  received  signal  and  reconstruct  a  64  x  64 
image  using  Eq.  (3). 

5  Results 

For  both  imager  configurations  the  reconstruction  was  per¬ 
formed  using  Matlab  software.  The  Matlab  Data  Acquisition 
Toolbox  was  used  to  acquire  the  data  from  the  data  acquisi¬ 
tion  device.  Code  was  written  to  implement  Eq.  (3)  and  to 
calculate  the  matrices.  The  disk  is  rotated  at  a  rate  of  50  rota¬ 
tions  per  second  and  therefore  allows  image  formation  at  50 
frames  per  second  (fps).  However,  the  data  acquisition  speed 
of  the  current  implementation  is  only  10  fps,  and  when  an 
image  is  reconstructed  and  displayed  on  the  screen  the 
image  rate  is  6  fps.  These  slower  frame  rates  are  due  to 
the  implementation  in  Matlab.  On  a  laptop  PC  (processor 
Intel(R)  Core(TM)2  Duo  CPU  T6670  at  2.2  GHz),  the 
data  reconstruction  step  (matriX[4094X4094j  vector  14096*  q  pro¬ 
duct)  in  the  current  Matlab -implementation  takes  only  p0  ms 
and  the  rest  of  the  loop  time  is  occupied  by  the  data  acquisi¬ 
tion,  image  reshaping,  filtering,  and  image  display.  All  these 
steps  can  be  made  faster  using  a  compiled  implementation 
and  dedicated  processing. 

For  the  transmission  mode  configuration  we  used  the 
mask  shown  in  Fig.  7(a)  as  the  object  An  aperture  in  the 
shape  of  a  hand  gun  was  cut  on  an  aluminum  sheet  and 
the  sheet  was  placed  in  front  of  the  coded  aperture  as 
shown  in  Fig.  5.  For  the  stand-off  configuration,  an  alumi¬ 
num  plate  was  roughened  by  sanding  and  a  piece  of  carpet 
with  an  aperture  in  the  shape  of  a  hand  gun  was  placed  over 
it.  This  target  is  shown  in  Fig.  7(b).  The  target  was  placed 
10  m  from  the  image  forming  mirror,  as  shown  in  Fig.  6,  and 
an  image  was  formed  on  the  coded  aperture. 

The  results  of  the  reconstruction  are  shown  in  Fig.  7(c) 
and  7(d).  In  both  cases  the  shape  of  the  gun  can  be  distin¬ 
guished.  The  image  formed  in  the  transmission  mode 
configuration  resembles  the  target  more  than  the  image 
formed  in  the  stand-off  configuration.  One  of  the  reasons 
for  this  difference  is  the  presence  of  an  optical  system  in 
the  stand-off  configuration,  which  amplifies  the  effects  of 
coherent  imaging  because  of  aberrations.  Another  reason 
is  the  specular  nature  of  active  imaging.  Both  contribute 
to  the  nonuniformity  of  the  image  intensity  on  the  image 
plane  for  the  stand-off  case.  We  note  that  in  order  to  acquire 
the  image  in  stand-off  mode  [Fig.  7(d)]  the  object  must  be 
positioned  precisely.  A  slight  angular  movement,  with 
respect  to  the  optical  axis,  causes  one  part  of  the  object  to 
be  imaged  with  higher  intensity  than  the  other  parts,  'and 
only  it  can  be  imaged. 

The  reconstruction  error  is  present  in  both  reconstruc¬ 
tions.  As  discussed  in  Sec.  3,  this  is  an  effect  of  the 
ill-conditioned  measurement  matrix  and  the  measurement 
noise,  the  nonlinearity  of  the  measurement,  and  the  assump¬ 
tion  of  static  hole  patterns  during  the  detector  integration 


(c)  <d) 


Fig.  7  Imaging  targets  and  image  reconstructions  from  each  config¬ 
uration;  (a)  target  mask  used  for  sub-millimeter  wave  transmission 
mode  configuration,  (b)  target  for  the  sub-millimeter  wave  stand-off 
mode  configuration,  (c)  reconstruction  from  the  sub-millimeter  wave 
transmission  mode  configuration,  (d)  reconstruction  from  the  sub¬ 
millimeter  wave  stand-off  mode  configuration. 

time.  The  reconstruction  noise  appears  in  the  form  of  clutter 
since  the  reconstruction  required  high  levels  of  regulariza¬ 
tion.  In  both  cases  the  norm  and  the  Laplacian  of  the  solution 
were  minimized  as  a  result  of  the  regularization. 

6  Conclusions 

We  have  built  an  image  plane  coded  aperture  device  for 
sub-millimeter  wave  imaging.  This  coded  aperture  has 
high  modulation  levels,  high  scanning  rates,  low  mechanical 
complexity,  and  low  cost.  The  design  parameters  were  iden¬ 
tified  in  our  earlier  publications.9,1^1  We  presented  a  math¬ 
ematical  formulation  for  the  imaging  problem  with  the  image 
plane  coded  aperture  and  a  regularized  least  squares  recon¬ 
struction  method  that  performs  in  real  time.  The  effect  of  the 
regularization  was  discussed  in  terms  of  the  relative  recon¬ 
struction  error  and  in  terms  of  the  nature  of  the  reconstruction 
error  or  noise.  Low  regularization  results  in  high  frequency 
spatial  and  temporal  noise,  while  high  regularization  results 
in  clutter  type  noise.  For  some  coded  aperture  configurations, 
e.g.,  the  two-dimensional  configuration,  high  regularization 
is  required  because  the  measurement  matrix  is  very  ill- 
conditioned.  For  other  configurations,  e.g.,  the  line  imager 
configuration,  low  regularization  is  possible  and  the  recon¬ 
structions  have  low  error.  The  imagers  that  we  implemented 
and  presented  here  are  two-dimensional  and  can  scan  images 
at  50  fps.  They  can  record  images  at  10  fps.  They  can  scan, 
reconstruct,  and  display  images  at  6  fps.  The  data  acquisition 
and  image  reconstruction  chains  are  currently  implemented 
on  a  personal  computer  using  Matlab  interpreted  code.  The 
processes  of  data  acquisition,  image  reconstruction,  and 
image  display  can  be  accelerated  to  produce  images  at  a 
rate  of  50  fps  if  the  code  is  compiled  and  a  dedicated 
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computer  is  used.  Our  next  efforts  will  be  focused  on  better 
models  of  the  measurement  matrix  and  the  building  of  a  line 
imager,  this  will  include  optics  appropriate  for  line  imaging. 
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Abstract:  Image  plane  coding  is  an  alternative  to  focal  plane  arrays  in  terahertz  imaging  devices.  A 
device  for  terahertz  image  plane  coding  is  described.  Analysis  shows  that  line  imaging  devices  are 
favored  as  opposed  to  two-dimensional  imaging. 
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1.  Introduction 

The  design  of  imaging  sensors  operating  at  millimeter  and  sub-rrullimeter  wavelengths  is  complicated  by  the  lack  of 
readily  available  focal  plane  arrays.  The  engineering  of  focal  plane  arrays  at  these  wavelengths  is  challenging  due 
to  either  cooling  requirements  for  passive  sensing  or  heterodyning  for  active  sensing[l,  2,  3].  To  date  imaging  in 
these  regimes  has  been  dominated  by  sensors  comprised  of  a  single  or  a  few  detectors.  In  order  to  construct  a  full 
size  image  a  mechanical  scanning  system  is  often  used.  Alternatives  involving  the  use  of  phased  array  techniques  are 
currently  being  developed  as  well[4].  Our  approach  to  this  problem  has  been  to  pursue  the  idea  of  image  plane  coding. 
Image  plane  coding  is  the  process  of  encoding  the  information  in  the  image  into  a  series  of  linear  measurements  over 
time  which  are  processed  to  reconstruct  the  image.  A  variety  of  devices  have  been  employed  for  visible  band  image 
plane  coding  including  liquid  crystal  spatial  light  modulators  and  digital  mirror  devices.  The  particular  device  we 
have  developed  is,  as  far  as  we  know,  the  only  device  that  is  capable  of  operation  at  millimeter  and  sub-millimeter 
wavelengths  and  that  allows  real  time  image  formation. 


2.  Measurements 


As  mentioned  in  the  introduction,  image  plane  coding  is  the  encoding  of  an  image  into  a  series  of  linear  measurements 
on  the  image.  Each  measurement  is  a  linear  combination  of  many  pixels  in  the  image.  Mathematically,  the  process  is 
described  by 

y  =  Mx  +  77  (1) 


where  x  is  a  vector  representing  the  image,  M  is  a  matrix  representing  the  measurement  process,  77  is  a  vector 
representing  the  noise  in  the  measurement  process,  and  y  is  a  vector  of  observed  measurements.  If  N  is  the  number 
of  pixels  in  the  image  x  and  M  is  the  number  of  measurements  in  y  then  M  is  a  M  x  N  matnx.  For  M  <  N, 
the  solution  to  equation  1  involves  iteration  and  hence  is  not  suited  for  real  time  operation.  In  addition,  images  of 
most  objects  of  interest  in  the  sub -millimeter  wave  range  will  be  limited  in  size  due  to  the  resolution  restrictions  of 
diffraction.  Therefore,  we  only  consider  the  case  of  M  =  N. 

Engineering  an  imaging  system  from  equation  1  requires  the  co-design  of  the  measurement  matrix  M  and  an 
algorithm  for  reconstructing  the  image  x  from  the  measurements  y.  Because  of  measurement  noise,  the  reconstructed 
image  will  only  be  an  approximation  of  the  real  image  x.  Ill-conditioning  in  the  measurement  matrix  amplifies  the 
noise  in  most  reconstruction  algorithms  and  hence  also  increases  the  uncertainty  in  the  estimate  of  of  image.  These 
considerations  are  addressed  in  inverse  methods  employing  regularization.  The  particular  method  we  have  to  date 
found  most  successful  with  our  method  of  generating  measurements  is  a  generalized  form  of  Tikhonov  regularization 
defined  by  the  following [5]. 

r  i-l 


x  = 


MTy  =  Ry 


(2) 


In  equation  2,  the  parameters  fii  are  Lagrangian  multipliers  or  the  regularization  parameters  and  the  matrices  Ht 
define  linear  operations  performed  on  the  results.  By  appropriate  choice  of  these  matrices  and  multipliers  the  effects 
of  noise  can  be  minimized  in  the  reconstruction.  For  our  work  we  use  two  linear  operators;  the  linear  operator  and  the 
Laplacian  operator.  Note  that  once  the  measurement  matrix  has  been  designed  and  the  linear  operators  selected,  the 
matrix  R  is  known  so  that  image  reconstruction  is  simply  a  matrix  multiplication. 


Fig.  1:  Schematic  representation  of  spinning  disk  image  plane  coding  apparatus. 


3.  Device 

The  device  we  use  for  generating  measurement  matrices  consists  of  a  spinning  metal  disk  with  a  pattern  of  circular 
holes  coupled  with  a  single  heterodyne  detector.  The  device  is  positioned  in  the  imaging  plane  of  an  optical  system.  The 
image  is  restricted  to  a  rectangular  section  of  the  disk  through  the  use  of  a  mask.  The  device  is  shown  schematically  in 
Figure  1.  Holes  in  the  metal  disk  are  used  for  sampling  the  image.  We  use  hole  radii  of  1.4 A  in  a  2.7A  thick  metal  disk 
and  achieve  a  transmission  of  approximately  100%.  In  order  to  achieve  independent  measurements  it  is  important  to 
avoid  hole  to  hole  coupling  effects.  We  have  found  that  a  hole  spacing  of  A  or  greater  achieves  this  goal.  It  should  be 
noted  that  a  hole  pattern  designed  for  any  wavelength  will  also  work  well  for  any  smaller  wavelength  meaning  that  the 
hole  pattern  will  preserve  the  transmission  properties  and  independence  properties  achieved  at  the  larger  wavelength. 

The  design  of  the  hole  pattern  and  the  sampling  scheme  of  the  image  and  detector  are  the  main  factors  governing 
the  properties  of  the  measurement  matrix.  Hole  patterns  are  constructed  by  placing  possible  hole  sites  in  a  hexagonal 
pattern  in  a  square  having  side  length  equal  to  the  diameter  of  the  disk.  The  side  length  of  each  hexagon  is  selected  to 
be  3. 8 A  which  will  allow  for  1.4A  radius  holes  at  A  spacing.  The  hexagonal  pattern  results  in  the  closest  hole  packing 
density.  By  selecting  an  occupation  probability  and  randomly  assigning  actual  holes  to  each  of  these  sites,  a  unique 
hole  pattern  is  generated.  Only  holes  within  an  annulus  defined  by  the  imaging  window  are  actually  fabricated  on  the 
disk. 

Once  a  hole  pattern  is  determined,  the  sampling  scheme  of  the  image  and  detector  can  be  used  to  form  the  measure¬ 
ment  matrix.  The  imaging  window  is  divided  into  virtual  pixels  of  rectangular  shape  (see  inset  in  Figure  1)  and  arbitrary 
size.  Each  pixel  is  assumed  to  have  a  constant  intensity  and  together  form  the  vector  x.  The  entries  in  a  single  row 
of  the  measurement  matrix  are  computed  as  the  fractional  area  of  each  visible  hole  on  each  pixel.  The  disk  is  rotated 
behind  the  imaging  window  to  produce  a  new  pattern  of  holes  in  the  imaging  window  and  another  row  in  the  measure¬ 
ment  matrix  is  generated.  This  continues  until  the  N  x  N  measurement  matrix  is  filled.  As  the  disk  rotates,  samples 
of  the  energy  transmitted  through  the  holes  are  collected  by  the  receiver.  If  the  disk  does  not  rotate  very  far  between 
samples  by  the  receiver,  there  could  be  significant  correlation  between  adjacent  rows  in  the  measurement  matrix.  The 
correlation  is  directly  dependent  on  how  long  a  hole  is  within  the  imaging  window.  Figure  2(a)  shows  the  transit  time 
of  a  single  hole  through  the  imaging  window  normalized  to  the  sampling  time  for  various  image  sizes.  Sampling  time 
is  determined  by  the  size  of  the  image  and  the  speed  of  rotation  of  the  disk.  Since  patterns  repeat  after  one  rotation 
of  the  disk,  to  avoid  correlations  due  to  repeated  patterns,  the  measurement  matrix  must  be  filled  in  one  rotation.  As 
a  result,  the  sampling  time  is  simply  the  number  of  pixels  in  an  image  N  divided  into  the  rotation  time.  Since  each 
rotation  yields  an  image,  the  rotation  time  is  the  frame  time  for  video. 

The  vertical  axis  of  Figure  2(a)  is  the  number  of  samples  taken  during  a  transit  time.  With  hundreds  of  samples 
being  taken  for  a  square  image  and  only  one  or  two  for  a  line  image  it  is  clear  that  avoidance  of  correlation  in  the 
measurement  matrix  will  favor  line  image  scenarios.  This  can  be  seen  more  clearly  by  viewing  measurement  matrix 
condition  numbers  plotted  against  normalized  transit  time  (Figure  2(b)).  This  Figure  was  produced  by  making  10  disk 
realizations  and  producing  measurement  matrices  for  64  x  K  images  where  K  =  1,2, 4, 8, 16,  32, 64.  The  condition 
numbers  for  each  realization  are  then  averaged  at  each  factor  of  K.  As  the  factor  K  increases,  the  logarithm  of  the 
condition  number  of  the  measurement  matrix  increases  in  a  power  law  fashion.  This  is  demonstrated  in  Figure  3  where 
simulations  of  the  measurement  and  reconstruction  process  are  done  for  both  two  dimensional  imaging  and  a  line 
imaging  scanner.  The  difference  between  the  two  dimensional  and  line  images  is  that  only  the  vertical  dimension  is 
reconstructed  in  the  line  image  whereas  both  are  reconstructed  in  the  two  dimensional  image.  In  practical  implemen¬ 
tations,  the  line  image  would  have  to  be  scanned  across  the  target  to  make  an  image. 


(a)  (b)  (c) 

Fig.  3:  Target  image  (a),  two-dimensional  imager  reconstruction  (b),  and  line  imager  reconstruction  (c) 


4.  Conclusions 

To  date,  we  have  constructed  both  line  imagers  and  two  dimensional  imagers  operating  at  640  GHz  and  optical  wave¬ 
lengths  using  this  approach[6,  7].  While  we  believe  we  understand  the  imaging  process  using  this  device  there  are 
several  avenues  for  improvement  that  are  being  pursued.  Our  primary  efforts  are  being  expended  on  optimizing  the 
measurement  matrixyhole  pattern  design.  Our  current  approach  is  trial  and  error  and  as  we  refine  our  models  of  the 
imaging  process  we  will  improve  the  procedure  for  generating  hole  patterns.  We  are  also  improving  the  optic  charac¬ 
teristics  of  our  system.  Further  efforts  at  characterizing  and  reducing  measurement  noise  are  also  being  undertaken. 
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Abstract — Image  plane  coded  aperture  (IPCA)  detectors  are 
composed  of  a  single  element  detector  sensitive  to  the  radiation 
frequencies  of  Interest,  a  re  configurable  spatial  light  modulator 
(SLM),  and  a  mechanism  such  as  a  lens  or  a  horn  that  Is  used 
to  collect  the  radiation  modulated  by  the  SLM  and  focus  It  on 
the  sensing  element,  The  IPCA  detector  Is  placed  hi  the  Image 
plane  of  an  imager  and  Is  used  to  make  linear  measurements  on 
the  image  by  modulating  the  signal  spatially  and/or  temporally 
in  amplitude,  frequency,  phase,  or  polarization.  The  Image  Is 
then  reconstructed  computationally.  In  this  paper  we  present 
an  Implementation  of  an  EPCA  detector  that  can  be  used  for 
Terahertz  Imaging  and  discuss  other  possible  Implementations. 

Index  Terms — Terahertz  imaging,  Coded  apertures,  Radar 
imaging 

I.  Introduction 

UCH  of  the  interest  in  terahertz  imaging  is  motivated 
by  the  abilities  of  THz  frequencies  to  penetrate  most 
manmade  materials  particularly  clothing  [1],  [2].  Additionally, 
many  harmful  chemicals  and  explosives  display  absorption 
lines  that  make  them  identifiable  by  THz  [3].  These  charac¬ 
teristics  of  THz  make  it  attractive  for  security  applications. 
Unfortunately,  focal  plane  arrays  for  this  regime  have  not  been 
commercially  forthcoming  and  current  offerings  either  lack 
sensitivity  or  are  too  costly  to  be  used  in  practical  security 
situations.  An  intermediate  solution  has  been  the  development 
of  scanning  systems.  These,  however,  have  their  own  cost, 
weight,  power,  and  frame  rate  limitations. 

Our  interest  is  focused  in  the  part  of  the  THz  spectrum 
between  300GHz  and  3THz  which  is  referred  as  the  sub¬ 
millimeter  wave  region.  In  this  region,  sensitive  detectors 
are  available  and  the  technology  is  mature.  Both  active  and 
passive  imaging  configurations  at  these  frequencies  have  been 
successfully  demonstrated.  Currently,  the  new  frontier  in  sub¬ 
millimeter  wave  imaging  is  faster  frame  rates  and  smaller  size, 
weight,  power  consumption,  and  cost.  Passive  systems  that 
operate  at  close  to  real-time  require  cooled  detectors  and  are 
large  [4],  [5].  Images  from  these  systems  are  similar  to  infrared 
images  providing  temperature  difference/contrast  information 
and  therefore  are  easy  to  interpret.  On  the  other  hand,  the 
images  acquired  with  active  imagers,  that  are  coherent,  have 
very  high  dynamic  range  and  suffer  from  the  specular  reflec¬ 
tions  and  clutter  in  the  scene,  which  makes  them  difficult  to 
interpret,  However,  information  from  active  THz  radar  imaging 
is  easy  to  interpret  [6].  An  advantage  of  active  systems  is  that 
they  operate  at  room  temperature  and  usually  their  physical 
size  is  limited  by  the  optical  aperture  rather  than  the  detector 
and  source  modules. 


One  of  the  most  promising  sub-millimeter  wave  systems  is 
the  imaging  radar  system  developed  by  NASA’s  Jet  Propulsion 
Laboratory  [6].  It  mitigates  clutter  and  specular  reflection  from 
the  scene  by  forming  range  maps  of  the  scene  rather  than  con¬ 
trast  images.  The  JPL  system  is  currently  limited  to  1Hz  frame 
rates.  The  reported  bottlenecks  are  the  mechanical  scanning 
and  the  signal  to  noise  (SNR)  of  single  pulse  detection.  The 
authors  of  that  work  acknowledge  the  practical  limitations  with 
speeding  the  mechanical  scanning  system  and  propose  the  use 
of  transceiver  arrays  as  a  solution  to  faster  frame  rate.  One 
alternate  approach  to  the  mechanical  scan  has  been  proposed 
by  Lukkanen  [7]  and  entails  using  an  electronically  controlled 
reflect- array  that  can  be  used  as  a  beam  steerer.  This  work 
is  in  its  design  phase.  Our  substitute  for  mechanical  scanning 
is  the  image  plane  coded  aperture  (IPCA)  detector.  In  this 
paper  we  present  the  concept  of  the  IPCA  detectors,  show  one 
implementation,  and  discuss  other  possible  implementations  of 
such  detectors. 

II.  Image  Plane  Coded  Aperture  Detectors 

Coded  aperture  detectors  are  usually  envisioned  as  substi¬ 
tutes  for  the  optical  system  of  an  imager  when  the  optical 
aperture  is  too  large  to  allow  the  use  of  optics.  Image  plane 
coded  aperture  detectors  on  the  other  hand,  are  used  as 
substitutes  for  focal  plane  arrays  in  frequency  regimes  where 
focal  plane  arrays  are  impractical,  expensive,  or  non-existent. 
A  conceptual  sketch  showing  the  place  of  the  IPCA  detector 
in  an  imaging  system  and  its  components  is  shown  in  Fig.  1. 


IPCA  detectors  are  composed  of  a  single  element  detector 
sensitive  to  the  radiation  frequency  of  interest,  a  reconfigurable 
spatial  light  modulator  (SLM),  and  a  mechanism  such  as  a 
lens  or  a  hom  that  is  used  to  collect  the  radiation  past  the 
SLM  and  focus  it  on  the  sensing  element.  The  IPCA  detector 
is  placed  in  the  image  plane  of  an  imager  and  is  used  to 


make  Linear  measurements  on  the  image  by  modulating  the 
information  on  the  image  plane  spatially  and/or  temporally 
in  amplitude,  frequency,  phase,  or  polarization.  The  image 
is  then  reconstructed  computationally  using  inverse  imaging 
techniques[8]. 

Because  the  IPCA  detectors  make  linear  measurements  of 
the  image  plane  a  computational  reconstruction  is  necessary. 
For  real-time  imaging  these  computations  should  happen  faster 
than  the  required  frame  rate.  Image  reconstruction  algorithms 
that  satisfy  this  criterion  are  closed  form  solutions  that  are 
reduced  to  simple  matrix  vector  multiplications.  However, 
because  of  other  practical  limitations  sub- millimeter  wave 
images  usually  have  low  resolution  (i.e.  64x64  pixels)  and 
iterative  solutions  can  also  be  used  without  compromising 
frame  rate.  The  IPCA  detectors  can  also  be  used  to  perfoim 
compressive  imaging  [9]. 

Depending  on  the  method  of  modulation  the  IPCA  detectors 
can  be  used  in  active,  passive,  coherent,  or  incoherent  modes. 
As  such,  the  use  of  the  detectors  is  not  limited  to  imaging 
but  can  also  be  used  in  imaging  radar,  spectroscopy,  or  multi¬ 
modal  imaging.  Multi-modal  imaging  is  facilitated  by  placing 
a  filer  in  front  of  the  detector  or  by  using  several  single  element 
detectors  responsive  to  particular  wavelengths.  Other  config¬ 
urations  can  be  envisioned  where  the  IPCA  detectors  operate 
in  coherent  and  incoherent  mode  (containing  a  coherent  and 
an  incoherent  single  element  detector)  to  form  contrast  images 
as  well  as  range  profiles  of  the  scene.  An  IPCA  detector  can 
also  be  used  to  make  linear  measurements  of  dispersed  spectra 
similar  to  Hadamard  spectroscopy  [10]. 

From  the  aspect  of  signal  to  noise  ratio  (SNR),  the  IPCA 
detectors  are  favored  by  active  imaging  configurations.  For 
similar  illumination  power,  the  detector  SNR  is  comparable 
to  a  conjugate  point  imaging  system  and  much  higher  than 
focal  plane  arrays  [11].  For  passive  systems  the  performance 
is  worse  than  that  of  the  focal  plane  arrays  but  comparable  with 
the  conjugate  point  system  [12].  However,  the  per  pixel  SNR 
after  the  image  is  reconstructed  computationally  suffers  by  an 
additional  factor  which  can  be  quite  high  if  the  modulation 
patterns  of  the  SLM  are  not  selected  properly, 

III.  Spinning  Disk  IPCA  Detector 

One  possible  realization  of  an  IPCA  detector  that  we  have 
built,  is  shown  in  Fig,  2,  The  SLM  consists  of  a  portion 
(labeled  as  imaging  window  in  Fig.  2)  of  a  spinning  disk 
with  random  holes.  For  each  pattern  of  holes  in  the  imaging 
window  (i.e.  a  code)  a  measurement  is  made  by  the  receiver 
and  it  is  registered  with  the  pattern.  If  the  holes  are  designed 
properly,  the  measurement  will  consist  of  the  linear  combi¬ 
nation  of  portions  of  the  image.  Once  the  desired  number  of 
measurements  are  made,  usually  a  number  equal  to  the  number 
of  desired  image  pixels,  the  hole  patterns  and  corresponding 
measurements  are  used  to  decode  the  image  formed  in  the 
imaging  window. 

The  insert  in  Fig.  2  and  equation  1  help  illustrate  this 
process.  The  imaging  window  is  divided  into  pixels  (n2  for  this 
example),  not  necessarily  square  or  regular,  For  each  pattern 
of  holes  or  code,  we  form  an  equation  where  the  weighted 


Figure  2.  Spinning  disk  IPCA  detector 

pixel  values  (pixel  values  are  the  unknowns)  add  up  to  the 
measured  value.  The  weights  represent  how  much  each  pixel 
contributes  to  the  measurement  and  are  calculated  as  the  ratio 
of  the  hole-pixel  intersect  area  to  the  total  area  of  the  pixel. 
If  the  pixels  are  of  equal  area,  then  the  equation  is  complete. 
If  they  are  not,  an  additional  factor  representing  the  relative 
sizes  of  the  pixels  is  computed  and  multiplied  with  the  other 
weights.  Using  the  equations  from  all  the  measured  patterns,  a 
system  of  equations  is  formed  and  solved  using  linear  algebra 
methods  (e.g.  regularized  linear  least  squares)  [8]. 

mllPl  +  ••  +  mlnpn  +  ••  +  mln*pn*  =  ml 

...  =  m.. 

+  "  H-mn2npn  +  ..  +  mn2n2pn3  =  mn2  (1) 

The  other  parts  of  the  IPCA  detector  are  the  collecting 
horn  and  single  element  detector.  We  use  a  Schottky  diode 
heterodyne  transceiver  pair  from  Virginia  Diodes  operating 
at  a  center  frequency  of  640  GHz.  One  possible  imaging 
configuration  that  we  have  constructed  in  our  lab  is  shown 
in  Fig.  3.  The  optical  system  consists  of  a  12  inch  diameter 
elliptical  aperture  with  one  focus  at  one  meter  (image  space) 
and  another  at  ten  meters  (target  space).  The  source  is  pointed 
directly  to  the  target  three  meters  away  and  the  optical  aperture 
is  approximately  ten  meters  away  from  the  target.  The  disk 
spins  at  50  rotations  per  second  which  corresponds  to  an  image 
frame  rate  of  50Hz  since  the  necessary  measurements  are  made 
during  one  revolution  of  the  disk.  The  measurements  out  of  the 
receiver  are  digitized  using  a  16-bit  data  acquisition  card  The 
card  is  read  by  a  custom  Matlab  script  which  also  performs 
the  image  reconstruction  and  display.  The  Matlab  part  of  the 
system  slows  down  the  image  formation  and  display  rate  to 
approximately  16Hz  when  run  on  a  laptop. 

Our  target  was  an  aluminum  plate  covered  by  carpet  con¬ 
taining  an  aperture  in  the  shape  of  a  gun  as  shown  in  Fig.  3. 
The  reconstructed  image  is  shown  to  the  right.  Three  bright 
spots  are  clearly  distinguishable,  the  handle,  middle  part,  and 
barrel.  The  clutter  in  the  image  is  due  to  the  reconstruction 
process  which  introduces  an  approximation  error  and  noise 
propagation  from  the  measurement  to  the  image.  The  disconti¬ 
nuity  of  the  target  image  is  due  to  the  nature  of  active  imaging, 
i.e.  specular  reflections  and  coherent  interference. 

Future  efforts  will  be  focused  towards  adapting  the  spinning 
disk  IPCA  detector  to  an  imaging  radar  system.  Also,  analysis 
has  shown  that  a  line  spinning  disk  IPCA  detector  results  in 
lower  approximation  error  and  noise  propagation  during  the 
image  reconstruction  process.  Another  one  of  our  efforts  will 


Figure  3.  (top)  stand-off  setup,  (bottom  left)  target,  (bottom  right)  corre¬ 
sponding  reconstructed  64x64  pixel  image  acquired  at  50  frames  per  second 
and  reconstructed  at  16  frames  per  second. 


be  the  development  of  a  line  IPCA  detector  coupled  with  a 
single  tilting  mirror  to  form  two  dimensional  images. 


IV.  Possible  IPCA  Detectors 

The  minimization  of  approximation  error  and  noise  propa¬ 
gation  during  the  image  reconstruction  process  is  strongly  de¬ 
pendent  on  the  code  implemented  by  the  detector.  An  optimum 
code  is  the  Hadamard  code  [10].  The  Hadamard  code  does  not 
propagate  noise  and  also  has  very  low  approximation  error 
since  it  would  not  require  large  amount  of  regularization[8]. 
The  Hadamard  code  is  composed  of  orthogonal  vectors  con¬ 
taining  values  of  1  and  -1.  One  possible  implementation  of  an 
IPCA  detector  that  realizes  the  Hadamard  code  would  consist 
of  180  degree  phase  modulators,  and  would  be  suited  for 
imaging  radar  systems.  A  180  degree  phase  shift  is  equivalent 
to  multiplying  the  incident  held  by  -1. 

A  group  from  the  University  of  Virginia  has  built  and 
demonstrated  a  broad-band  180  degree  phase  shifter  using 
integrated  sub -millimeter- wave  Schottky  diodes  that  operates 
at  220GHz  with  a  bandwidth  of  55GHz  within  a  phase 
tolerance  of  +/-  15  degree[13].  The  researchers  claim  that  the 
technology  is  scalable  to  sub-millimeter  waves  in  the  640GHz 
region.  If  this  is  possible,  these  phase  shifters  would  be  ideal 
to  implement  air  to  air  couplers  that  can  serve  as  the  SLM 
part  of  the  IPCA  detectors.  They  would  allow  implementation 
of  Hadamard  codes  for  imaging  radar. 

V.  Conclusions 

We  have  introduced  the  concept  of  an  image  plane  coded 
aperture  (IPCA)  and  briefly  discussed  some  of  its  applications. 
An  IPCA  detector  that  utilizes  a  spinning  disk  with  holes  as  an 
SLM,  was  also  described  and  the  image  of  a  gun-like  target 
acquired  with  this  IPCA  detector  was  shown.  The  spinning 
disk  IPCA  scans  the  image  plane  at  50Hz.  The  image  quality 
with  this  system  suffers  from  approximation  error  and  noise 
propagation  artifacts  that  are  introduced  during  the  image 
reconstruction  process.  These  artifacts  can  be  eliminated  if 
a  Hadamard  coding  scheme  is  used.  We  give  examples  of 


existing  research  that  could  lead  to  a  device  for  implementing 
a  Hadamard  code  with  application  to  a  THz  imaging  radar. 
The  use  of  an  IPCA  detector  is  an  alternative  to  mechanical 
raster  scanning  of  antennas  or  mirrors  and  focal  plane  arrays. 
When  compared  with  scanning  an  imaging  antenna  or  mirror, 
the  IPCA  detector  has  faster  scan  rates,  and  possibly  low  cost 
and  compact  form  factor. 
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ABSTRACT 

Various  schemes  for  active  imaging  require  different  allocations  of  source  power  and  can  result  in  different  overall 
signal  to  noise  ratios.  At  the  University  of  Memphis  we  have  developed  an  image-plane  scanning  device  used 
with  a  single  pixel  detector  to  form  video  rate  images  of  the  scene.  Imaging  with  this  device  requires  flood 
illumination  of  the  scene.  Because  sub-millimeter  wave  sources  typically  produce  low  power,  it  is  a  common 
belief  that  flood  illumination  results  in  low  detected  signal  power  and  therefore  low  signal  to  noise  ratios  (SNR) 
at  the  detector.  In  this  work  we  quantify  the  SNR  at  the  detector  for  our  system  and  compare  it  to  conventional 
imaging  systems,  conjugate  point  imaging  systems,  and  focal  plane  array  imaging.  Unlike  the  other  two  systems, 
imaging  with  our  device  requires  an  additional  pixel  formation  step;  therefore,  the  SNR  at  the  detector  is  not 
the  per-pixel  SNR.  We  present  the  limits  of  the  per-pixel  SNR  and  discuss  its  dependence  on  various  device 
components. 

Keywords:  Sub- mi  Hi  meter  waves,  terahertz,  imaging,  spatial  light  modulator,  spatially  selective  mask,  com¬ 
pressive  sensing 


1.  INTRODUCTION 

In  general,  an  active  imaging  device  consists  of  an  optical  system,  an  illumination  system,  a  detector  system, 
and  a  signal  processing  system  for  image  reconstruction  and  display.  There  are  multiple  ways  these  systems  can 
be  configured  and  the  burden  each  one  bears  in  the  image  formation  process  depends  on  the  configuration.  One 
of  the  simplest  schemes  is  to  use  a  lens  or  mirror  to  focus  the  image  of  the  illuminated  scene  onto  a  focal  plane 
array  of  detectors.  Signal  processing  for  this  configuration  is  usually  simple  as  well  as  each  detector  in  the  focal 
plane  array  is  mapped  to  an  image  pixel.  For  sub-millimeter  wave  imaging,  focal  plane  arrays  of  the  desired 
sizes  are  either  expensive  or  non-existent.  Designers  of  imaging  systems  are  then  forced  to  look  at  different 
schemes  for  forming  images.  Almost  all  involve  trading  image  acquisition  time/detector  integration  time  for 
spatial  resolution.  Also  the  optimal  illumination  scheme  for  each  configuration  is  different.  As  a  result,  signal  to 
noise  ratios  for  such  systems  are  different  from  the  focal  plane  array  system. 

In  this  paper  we  examine  the  signal  to  noise  ratio  (SNR)  for  the  various  schemes  and  compare  them.  We 
restrict  our  study  to  active  imaging  where  image  brightness  (signal)  is  controlled  by  the  strength  of  the  source  over 
an  imaged  pixel.  Five  configurations  are  investigated,  namely:  conjugate  point  imaging,  conjugate  line  imaging, 
conjugate  plane  imaging,  conjugate  line-point  imaging,  and  conjugate  plane-point  imaging.  These  configurations 
are  described  in  more  detail  in  the  next  section,  but  the  conjugates  referred  to  in  the  system  nomenclature  are 
the  illuminated  region  in  the  object  plane  and  the  detector  array  dimension  in  the  image  plane.  The  last  two 
configurations,  conjugate  line-point  and  plane-point,  require  non-trivial  processing  of  the  measured  noisy  signal 
to  reproduce  the  image  formed  on  the  image  plane.  The  additional  processing  introduces  noise  in  the  final 
reconstructed  image,  therefore  these  two  systems  require  additional  analysis. 

In  the  next  section  we  describe  each  of  the  active  imaging  configurations  that  will  be  analyzed  and  compared. 
Section  3  provides  a  summary  of  the  derivation  of  SNR  of  the  detected  signal  for  active  heterodyne  systems 
as  presented  by  Brown.1  The  following  Sections  4  and  5  provide  an  analysis  of  each  imaging  configuration 
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with  respect  to  incident  power  on  the  detector  and  available  integration  time  per  measurement.  In  Section  6  we 
summarize  the  analysis  of  the  previous  sections  to  compare  the  SNR  at  the  detector  for  each  of  the  configurations. 
The  analysis  of  the  additional  reconstruction  noise  for  the  last  two  configurations  is  presented  in  Section  7  In 
Section  8  we  make  some  final  remarks  and  present  our  conclusions. 

2.  ACTIVE  IMAGING  CONFIGURATIONS 

Active  imagers  consist  of  an  illuminating  source,  illumination  optics,  image  forming  optics,  and  the  detector. 
The  detector  can  be  a  single  or  multi-element  detector,  and  more  generally  it  can  have  a  spatial- temporal  varying 
integration  aperture.  As  is  usually  the  case,  the  imaging  and  illumination  optics  are  the  same,  and  the  source 
and  detector  are  multiplexed  using  a  beam  splitter.  Depending  on  the  imager  configuration,  the  optics  may 
contain  a  beam  steering  mechanism.  The  illuminating  source  might  illuminate  over  regions  of  various  shapes 
and  usually  consists  of  a  single  point  source  and  beam  shaping  elements. 

Figures  la  through  le  show  five  possible  active  imaging  configurations.  Figure  la  shows  a  conjugate  point 
imaging  configuration.  In  this  configuration  the  illumination  source  illuminates  a  diffraction  limited  spot  in  the 
object  plane  and  the  same  spot  is  imaged  on  a  single  element  detector  in  the  image  plane.  Spatial  resolution  is 
achieved  by  means  of  a  scanning  mechanism  that  concurrently  scans  the  illumination  source  and  single  element 
detector. 

Figure  lb  shows  a  conjugate  line  imaging  configuration.  Here  the  illumination  source  is  projected  over  a 
line  segment  in  the  object  plane;  the  part  of  the  object  that  is  illuminated  is  imaged  in  the  image  plane  on  a 
multi-element  detector  arranged  in  a  line  array  configuration.  One  dimension  is  resolved  by  the  multi-element 
array  and  the  other  by  concurrently  scanning  the  line  source  and  detector  across  the  scene. 

Figure  lc  shows  a  conjugate  plane  imaging  configuration.  The  illumination  source  is  projected  over  a  rect¬ 
angular  area  in  the  object  plane;  the  part  of  the  object  that  is  illuminated  is  imaged  in  the  image  plane  on  a 
multi-element  detector  in  a  rectangular  array  configuration.  Both  dimensions  are  resolved  by  the  positioning  of 
the  detectors  on  the  array.  This  configuration  does  not  require  any  scanning. 

Figure  Id  shows  a  conjugate  line-point  imaging  configuration.  Here  the  illumination  source  is  projected  over 
a  line  segment  in  the  object  plane;  the  part  of  the  object  that  is  illuminated  is  imaged  in  the  image  plane  on 
a  single  element  detector  with  a  line-shaped  spatial-temporal  varying  integration  aperture.  One  dimension  is 
resolved  by  varying  the  integration  aperture  of  the  detector  over  time  and  the  other  by  concurrently  scanning 
the  line  source  and  detector  across  the  scene.  The  variable  integration  aperture  is  usually  implemented  by  a 
mask  placed  over  the  detector  aperture.  As  shown  in  the  figure,  the  detector  includes  a  collecting  lens  and  the 
mask.  The  lens  is  used  to  optically  increase  the  integrating  aperture  of  the  detector.  Because  the  mask  only  lets 
part  of  the  radiation  incident  on  the  aperture  to  reach  the  receiving  detector  we  define  a  fill  ratio  constant,  fr. 
This  constant  represents  the  fraction  of  incident  energy  that  reaches  the  receiving  detector. 

Figure  le  shows  a  conjugate  plane-point  imaging  configuration.  Here  the  illumination  source  is  projected 
over  a  rectangular  area  in  the  object  plane.  The  part  of  the  object  that  is  illuminated  is  imaged  in  the  image 
plane  on  a  single  element  detector  with  a  rectangular-shaped  spatial- temporal  varying  integration  aperture.  Both 
dimensions  are  resolved  by  varying  the  integration  aperture  of  the  detector  over  time.  This  configuration  does 
not  require  any  image  or  detector  scanning. 

3.  SIGNAL  TO  NOISE  RATIO  OF  ACTIVE  HETERODYNE  SYSTEMS 

In  terms  of  SNR,  the  variables  of  interest  for  each  configuration  are  integration  time  and  power  incident  on  the 
detector  elements  for  each  measurement.  Before  examining  the  integration  time  and  incident  power  available 
for  each  of  the  five  configurations,  we  present  the  dependence  of  the  SNR  on  these  two  variables.  This  is  a 
summary  of  Brown’s1  SNR  derivation  for  an  active  heterodyne  system.  A  block  diagram  of  the  heterodyne 
receiver  analyzed  by  Brown  is  shown  in  Figure  2. 

Brown  defines  the  power  SNR  as  the  ratio  of  the  average  power  to  the  rms  fluctuations  associated  with  the 
noise 


(a)  Conjugate  point  imaging 


(b)  Conjugate  line  imaging 


(c)  Conjugate  plane  imaging 


(d)  Conjugate  line-point  imaging  (e)  Conjugate  plane-point  imaging 

Figure  1.  Imager  configurations 


Figure  2.  Block  diagram  of  the  heterodyne  system 
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where  SP  is  the  power  spectral  density  of  the  noise  and  Bjsnb  is  the  equivalent  noise  bandwidth  at  that  point 
in  the  sensor. 


Square-Law  detectors: 

A  square-law  detector  produces  an  output  proportional  to  its  square 


X out  —  AX?n  —  7 ZPaba 


(2) 


For  our  detectors  Xout  is  usually  a  voltage,  Pabs  is  the  power  absorbed  by  the  detector,  and  7 Z  is  the  detector 
responsivity  in  units  of  V/W.  Assuming  that  the  input  noise  to  the  detector  has  white  noise  spectrum  and  is 
unrelated  to  the  signal,  the  SNR  at  the  output  of  the  detector  is  related  to  the  SNR  at  the  input  by 


SNR^t 


SNRin 
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where  Ai^  is  the  signal  (noise)  bandwidth  before  detection  and  A /  is  the  signal  (noise)  bandwidth  after  detection. 
The  bandwidth  after  detection  A/,  is  determined  by  the  low-pass  filter/integrator  following  the  detector.  The 
integration  time  of  the  detector  (also  determined  by  the  low-pass  filter/integrator)  is  r  =  1/(2A/),  therefore  the 
SNR  relationship  above  can  be  expressed  as 

SNRout  =  SN^JTTu  (4) 


Heterodyne  conversion: 

For  the  heterodyne  conversion  the  output  signal  (a  voltage)  is  equal  to  the  product  of  the  input  signal  and 
local  oscillator  signal  (LO)  times  a  proportionality  constant,  A.  Assuming  sinusoidal  signals  ( X{n  =  Bcos(u>int) 
and  Xio  =  Ccos(u>Lot)  ),  the  output  signal  at  the  intermediate  frequency  (IF)  which  is  the  difference  of  the 
input  signal  frequency  and  LO  frequency  will  be 

Xout  =  ABCcos(<jJin  —  U>Lo)t  =  2 rfJl\/ PincPLOCOs(oJin  —OJLo)t  (5) 

where  Pinc  and  Plo  are  the  incident  and  LO  powers,  and  r/  is  the  fraction  of  each  that  is  absorbed  (mixer 
efficiency).  Then  the  output  power  at  the  IF  frequency  is  given  by 

Pout  =  DX^  =  4 D (rfFL) 2 Pinc Pl o cos 2 (u>in  -  ULo)t  =  2 D(rfJl)2PincP Lo  (6) 

where  D  is  the  mixer-to-IF  circuit  matching  factor  accounting  for  mismatch  between  the  mixer  and  IF  load 
impedance  and  the  bar  represents  time  averages.  Grouping  mixer  and  detector  related  terms  together  ( GmiX  = 
2DT12Plo)  j  the  IF  power  is  rewritten  as 


Pout  =  V2OmiXPinC  (7) 

The  noise  consists  in  the  fluctuations  of  the  mixer  output  signal  that  are  caused  by  quantum- mechanical 
fluctuations  in  the  incident  and  LO  signals.  Since  usually  Plo  Pina  the  local  oscillator  noise  will  dominate 
and  an  upper  limit  on  the  noise  (similar  to  direct  detection  noise,  since  the  mixer  is  a  direct  detection  detector 
with  two  arms)  will  be 

(AXotlt)2  =  2 n  ■  (A PLO)2  ■  ^  (8) 

where  Bjp  is  the  IF  bandwidth  (equivalent  to  the  after  detection  bandwidth)  and  Av  is  the  bandwidth  over 
which  the  mixer  accepts  radiation.  The  LO  quantum- mechanical  noise  power  is  given  by 


A  Plo  =  Vv'h-  vloPlo(A  v) 


(9) 


where  h  is  Plank’s  constant  and  vlo  =  w lo /27r.  The  noise  will  therefore  be 

(AA^t)2  =  2P  ■  7]  •  h  •  vloPloBif 

or  in  terms  of  noise  power 

AP&ut  =  T>(AA'out)2  =  2D7£  •  77  •  h  ■  vloPloBif  —  Gm^rj  •  /i  •  i'loPjf 
The  SNR  after  the  heterodyne  conversion  will  be 
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Electrical  noise  on  heterodyne  mixers: 

The  electrical  noise  introduced  by  the  IF  electronics  is  accounted  for  by  adding  a  noise  rms  term  to  the  IF 
noise.  We  call  the  resulting  noise  the  heterodyne  noise  and  the  resulting  SNR  the  heterodyne  SNR 
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The  electrical  noise  term,  yJjKPjp  «)2,  will  be  expressed  in  terms  of  the  mixer  noise  equivalent  power  (NEP). 
The  NEP  is  found  by  setting  the  SNR  of  the  system  to  unity  and  solving  for  the  incident  power,  therefore 


NEPHBT  =  J>_  =  h^ioilL  +  y(A^r-‘)! 
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The  specific  NEP  (NEP  per  unit  bandwidth)  is  obtained  by  dividing  the  NEP  by  the  signal  (noise)  bandwidth 

NEPhet  _  h  •  uio  , 


NEP HET  — 


Bjf 


+ 


V2GmixBlF 


(15) 


The  first  term  to  the  right  of  the  equation  is  due  to  the  quantum-mechanical  noise  of  the  mixer  and  the  second 
term  is  due  to  the  electrical  noise.  If  the  electrical  noise  is  much  larger  than  the  quantum-mechanical  noise 
and  the  electrical  noise  is  all  white  Gaussian  noise  (AWGN),  the  specific  heterodyne  NEP  reduces  to  the  mixer 
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therefore  we  can  solve  for  the  electric  noise  rms  term 


^/(AP,F_e)2  =  n2GmixBTF  ■  NEPmix 
The  heterodyne  SNR  of  Equation  13  is  then  rewritten  as 
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Rom  this  equation  we  also  obtain  the  specific  heterodyne  NEP  as  a  function  of  the  specific  mixer  NEP 
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This  form  of  the  specific  heterodyne  NEP  is  convenient  because  it  is  used  to  define  the  receiver  noise  temperature 
which  is  usually  specified  by  the  manufacturer  of  the  receiver.  The  receiver  noise  temperature  is  also  known  as 
noise- equivalent  temperature  (NET) 

NET„Er  =  ?^  =  ±(^  +  NEPm«,r)  (20) 

where  kg  is  the  Boltzmann  constant.  Given  the  receiver  noise  temperature,  NEPmix&r  can  be  calculated  and 
then  can  be  obtained.  For  example  for  our  640GHz  system,  i/lo  =  319 GHz,  NEThet  =  3000if, 

NEPmixer  =  4.12  xlO-'*[W], 

Heterodyne  detection  with  Schottky  diode  mixers: 

The  SNR  after  the  heterodyne  conversion  including  the  contribution  of  the  mixer  electrical  noise  is  given  by 
Equation  18.  Each  term  (signal  in  the  numerator  and  noise  in  the  denominator)  is  multiplied  by  the  gain  of  the 
entire  IF  amplifier  chain  Gjp,  and  the  responsivity  of  the  square  law  detector  7££>j,that  Is  used  to  detect  the 
amplified  IF  signal  .  The  relationship  of  the  SNRs  before  and  after  detection  for  square  law  detectors  is  given 
by  Equation  3  and  is  restated  here  with  the  appropriate  subscripts 


SNRad  =  SNRBD]J (2i) 

where  the  subscript  “BD”  stands  for  ‘"before  detection”  and  the  subscript  “AD”  stands  for  “after  detection”.  In 
our  case  Bed  is  Bjp ,  and  Bad  is  the  bandwidth  of  the  integrator, A/,  after  the  square  law  detector  .  The  SNR 
after  detection  then  becomes 
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Therefore  the  heterodyne  detection  the  SNR  is  directly  proportional  to  the  incident  power  and  inversely  related 
to  the  square  root  of  the  detection  bandwidth,  A/.  In  terms  of  incident  power  and  integration  time  the  detected 
SNR  is 
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where  the  integration  time  is  given  by  r  =  1/(2A/). 
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4.  INCIDENT  POWER 

Now  that  the  SNR  dependence  on  the  incident  power  and  integration  time  has  been  stated,  we  analyze  the  amount 
of  incident  power  on  the  detectors  of  each  imager  configuration.  This  analysis  is  simplified  by  introducing  the 
concept  of  spatial  modes.  The  number  of  spatial  modes,  M$/d^  of  the  source  or  detector  element  is  defined  as 
the  ratio  of  squares  of  the  sensor  (including  optics)  field  of  view  ( FOVsensor )  to  the  source/detector  element 
instantaneous  field  of  view  ( IFOVs/d );  equivalently,  the  ratio  of  the  solid  angles.  The  receiver  and  transmitter 
might  also  include  optics  or  other  components. 

For  conjugate  point  imaging,  source  and  detector  element  number  of  modes  is 


Ms  =  Mq  = 
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This  signifies  that  the  source  is  projected  over  the  region  of  interest,  and  that  region  of  interest  is  imaged  by  the 
detector  element. 


For  conjugate  line  imaging 
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where  n  is  defined  as  the  ratio  of  the  desired  linear  image  size  (in  the  image  plane)  to  the  square  root  of  the 
receiving  antenna  effective  area  (assuming  a  square  antenna  aperture).  The  effective  area  of  an  aperture  antenna 
can  be  less  than  or  equal  to  the  physical  area  of  the  antenna  aperture.2  We  can  use  the  upper  limit  (max 
fill  factor  100%)  and  assume  the  effective  and  physical  areas  to  be  equal.  This  can  also  be  interpreted  as  the 
number  of  pixels  in  the  linear  direction  for  100%  fill  factor.  The  number  source  modes  is  again  one  because  the 
illumination  source  in  this  case  is  a  line  segment  and  is  projected  over  the  region  of  interest.  The  number  of 
detector  element  modes  however  is  n  because  each  of  the  detector  elements  only  images  a  fraction  of  the  line 
segment. 


For  conjugate  plane  imaging  the  reasoning  is  similar  except  that  each  detector  element  only  images  1/n2  part 
of  the  image 
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where  the  image  is  assumed  to  be  square. 

For  conjugate  line-point  imaging  we  have 


FOVscnsor\  _  , 
IFOVd  J  U 


(29) 


Ms  = 


(FOVsensor) 

v  IFOVs  J 


=  1 


(30) 


Md  = 


(FOVsensor) 

v  IFOVd  J 


Sr 


(31) 


where  fr  is  the  mask  aperture  fill  ratio  defined  previously,  usually  a  value  less  than  one.  The  mask  aperture 
fill  ratio  represents  the  fraction  of  the  detector  aperture  that  is  transmissive  for  any  pattern.  Depending  on  the 
mask  configuration,  practical  values  for  this  constant  range  between  1/n2  and  0.7.  For  this  case  the  detector 
element  is  dependent  only  oil  the  fill  ratio  because  the  image  of  the  illuminated  object  region  is  formed  on  the 
detector  aperture  which  is  then  masked  with  a  fill  ratio  /r.  For  conjugate  plane-point  imaging  we  also  have  the 
same  relationships. 


The  number  of  spatial  modes  for  the  source  is  one  (Ms  =  1)  in  all  the  configurations  because  it  is  assumed 
that  the  source  power  is  perfectly  projected  on  the  target  by  the  optical  system,  ignoring  diffraction  effects.  If 
diffraction  effects  are  accounted  for,  the  number  of  spatial  modes  becomes  larger  than  one.  The  factor  of  spatial 
mode  increase  depends  on  the  F#  of  the  system’s  optical  system  and  the  area  of  the  illuminated  target  space.  As 
the  F#  increases,  the  spatial  mode  number  increases;  illumination  projected  onto  the  target  is  not  as  selective 
and  the  imaged  illumination  falls  outside  the  aperture  of  the  detector.  As  the  illuminated  area  increases,  the 
spatial  mode  number  approaches  one;  in  this  case  the  blurred  edge  to  total  illuminated  area  ratio  decreases  as 
the  area  increases.  Therefore,  accounting  for  diffraction  affects  the  conjugate  point  imaging  configuration  more 
than  the  others.  For  the  purposes  of  the  following  analysis  we  will  initially  ignore  diffraction  because  its  effect  is 
usually  less  than  an  order  of  magnitude;  but  we  will  pick  up  the  discussion  when  the  various  configurations  are 
compared  in  Section  6. 

The  relationship  between  source  power  and  incident  power  on  the  detector  is  calculated  using  the  Driis  formula 
found  in  many  books  on  radar  systems  or  antennas.2  Because  the  source  is  coupled  to  an  optical  system  and  it  is 
assumed  that  the  optical  aperture  and  source  antenna  beam  width  are  matched,  all  the  power,  attenuated  by  the 
atmospheric  loss  coefficient  (r(r))  and  the  reflectivity  or  transmission  of  the  optics  (cLoPtic*)>  Is  projected  on  the 


target.  This  is  true  in  all  of  the  configurations.  If  the  target  is  assumed  larger  than  its  illuminated  sections,  the 
reflected  power  will  be  equal  to  the  power  density  on  the  target  times  the  radar  cross  section  of  the  target  section 
and  attenuated  by  the  reflectivity  of  the  target  ( Rrarget )•  For  simplicity,  and  since  the  analysis  is  relative,  we 
assume  that  the  target  radar  cross-section  is  equal  to  the  illuminated  target  area,  and  any  losses  associated  with 
target  orientation  or  roughness  are  absorbed  into  the  reflectivity  coefficient.  The  power  reflected  by  the  target 
will  be: 

F Target  =  <^Optics'r(r)^Targe.t^) source  (32) 


The  power  collected  by  the  optical  aperture  is  then  the  power  density  at  the  aperture  due  to  the  reflecting 
target  (again  attenuated  by  the  atmosphere),  multiplied  by  the  physical  area  of  the  optical  aperture  (^optica)- 
It  is  assumed  that  the  optical  aperture  is  large  with  respect  to  wavelength  and  matched  to  the  receiving  antenna. 
The  power  collected  by  the  optics  is  then  projected  in  the  image  plane,  and  the  power  integrated  across  the 
image  plane  is 


p  _  Pr  ar  get?  (r'jAoptics&Opt  ics 

“  4^2 

D  _  PTargttPicmrctr^  {T^^Optics^Optics 


image. 


47 XT2 


(33) 


The  power  incident  on  the  receiving  antenna  will  be  a  function  of  the  power  in  the  image  plane  and  the  spatial 
mode  number  of  each  configuration 

<34> 


5.  INTEGRATION  TIME 


Integration  time  for  the  scanning  configurations  is  a  function  of  the  picture  elements  (pixels)  required  for  the 
image.  The  number  of  pixels  may  or  may  not  be  equal  to  the  number  of  spatial  modes.  For  staring  configurations 
integration  time  is  independent  of  the  number  of  pixels.  Image  frame  rate  affects  both  staring  and  scanning 
systems. 

For  conjugate  point  imaging  the  integration  time  will  be 


1 

F  x  (7A)2 


-T 


(35) 


where  F  is  the  frame  rate  in  Hz,  N2  is  the  number  of  pixels  per  image  or  the  total  number  of  measurements  per 
image,  7  is  the  pixel  oversampling  factor,  and  T  is  a  scanning  delay.  The  oversampling  factor  is  usually  greater 
or  equal  to  one. 

For  conjugate  line  imaging  the  integration  time  will  be 


T  Fx'jN ' 

where  it  is  assumed  that  the  pixelated  images  will  be  square. 
For  conjugate  plane  imaging  the  integration  time  will  be 
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(36) 


(37) 


For  conjugate  line- point  imaging  and  conjugate  plane-point  imaging  the  integration  time  will  be  the  same 
except  that  the  scanning  delay  terms  might  be  different 


FxN2 


-T 


(38) 


For  these  configurations  however,  N2  is  also  interpreted  as  the  number  of  measurements  which  can  be  less  than 
the  number  of  image  pixels. 

Depending  on  the  implementation  of  each  imaging  technique  the  number  of  image  pixels  or  measurements, 
A2,  and  the  number  of  effective  antenna  areas  in  the  image  plane,  n2,  may  or  may  not  be  equal. 


Table  1.  Relative  SNR  expressions  for  each  configuration 


Imaging  configuration 

Relative  SNR 

conjugate  point  imaging 

SNRPoint  —  ^  fx^n)7  ^ 

conjugate  line  imaging 

SNR'Line  =  dimJL 

conjugate  plane  imaging 

SNRplant  = 

conjugate  line-point  imaging 

SN Rune-Point  “  fr 

\J  FxN 2  T 

conjugate  plane-point  imaging 

SN  ^Plane-Point  =  fr 

\J  FxN2  ^ 

6.  RELATIVE  SNR 

Given  the  dependence  of  the  SNR  on  the  incident  power  and  integration  time,  and  the  expressions  for  those 
variables  in  terms  of  number  of  pixels,  frame  rate,  and  imager  configuration,  we  define  the  relative  SNR  for  each 
configuration.  The  relative  SNR  is  the  SNR  after  detection  (given  in  Equation  24)  normalized  by  all  the  common 
factors  (common  between  imaging  configurations) 


SNR  = 


SNRad 


{  rj'h'U 


loW-NEP, 'mix 

Using  Equations  24  and  34  in  39  we  can  express  the  relative  SNR  as 


c  image 


SNR  = 


Md 


(39) 


(40) 


Using  the  detector  mode  number  and  integration  time  expressions  derived  previously  we  find  the  relative 
SNR  expressions  for  each  configuration.  The  expressions  are  shown  in  Table  1. 

An  interesting  observation  is  that  conjugate  plane-point  and  line-point  imaging  perform  similarly  (differing 
in  the  scan  delays  and  mask  fill  factors).  For  typical  values  of  F  =  10 Hz}  n  =  1.2 N,  T  =  20%  delay,  fr  =  0.5, 
and  7  =  1,  we  plot  the  relative  SNRs  as  a  function  of  the  number  of  pixels  in  the  linear  direction,  N.  Figure  3a 
shows  plots  of  the  relative  SNRs  as  given  in  Table  1.  The  units  of  this  plot  are  arbitrary.  The  best  performance  is 
achieved  by  the  conjugate  point  imaging.  The  next  best  performance  is  achieved  by  the  conjugate  line-point  and 
conjugate  plane-point  systems.  The  conjugate  line  system  follows  and  the  worst  performing  system,  in  terms  of 
SNR,  is  the  conjugate  plane  system.  These  results  confirm  Grossman’s  result  that  active  systems  favor  scanning 
systems  with  regard  to  sensitivity.3  The  plots  also  show  that  the  SNR  decreases  as  the  number  of  image  pixels 
increases  for  a  set  frame  rate.  To  compare  the  rate  of  decrease  with  respect  to  number  of  image  pixels  we  plot  the 
data  from  Figure  3a  normalized  by  the  relative  SNR  of  the  conjugate  system.  These  plots  are  shown  in  Figure 
3b.  The  rate  of  decrease  of  the  conjugate  point,  conjugate  line-point,  and  conjugate  plane-point  is  the  same, 
while  the  other  two  systems  suffer  a  faster  decrease.  This  is  expected  as  the  relative  SNR  for  the  first  group  of 
systems  depends  only  on  the  integration  time  while  for  the  second  group  it  depends  on  the  integration  time  and 
number  of  pixels  and  modes. 

The  conjugate  line-point  and  plane  point  systems  can  also  be  used  in  a  compressive  sensing  mode,  where  the 
number  of  measurements  required  to  reconstruct  an  image  can  be  a  fraction  of  the  number  of  pixels  in  the  image. 
This  is  achieved  if  the  measurements  satisfy  certain  conditions  and  the  image  is  sparse  in  some  domain.4  The 
implication  is  that  by  decreasing  the  number  of  measurements  we  can  account  for  the  loss  due  to  the  mask  fill 
ratio  and  achieve  the  relative  SNR  of  the  conjugate  imaging  configuration  or  better.  Also,  if  diffraction  effects 
are  considered  in  the  formulation  of  the  mode  equations  in  Section  4,  the  performance  of  the  conjugate  point 
system  will  decrease,  possibly  falling  below  that  of  the  conjugate  line-point  and  plane-point  systems.  However, 
as  it  is  shown  in  the  next  section,  these  two  systems  suffer  by  additional  noise  that  is  introduced  during  the 
image  reconstruction  process. 
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Figure  3.  Relative  SNR  plots 
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7.  IMAGE  RECONSTRUCTION  “NOISE” 

The  association  of  measurements  with  digitized  pixels  on  a  screen  can  be  expressed  mathematically  as 


Mp  +  rj  =  m  =  m  +  rj  (41) 

where  M  is  a  matrix  that  relates  the  digitized  pixels  contained  in  the  vector  p  to  the  noisy  measurements  m.  The 
noisy  measurements  are  composed  of  the  signal  m  and  noise  rj.  The  matrix  M  is  called  the  measurement  matrix 
and  is  usually  calculated  based  on  assumptions  on  the  imaging  system.  For  example,  for  the  conjugate  point, 
line,  and  plane  systems  the  measurement  matrix  can  resemble  a  diagonal  matrix  that  may  or  may  not  model 
the  point  spread  function  of  the  optical  system.  For  the  conjugate  line-point  and  plane  point  the  measurement 
matrix  will  depend  on  the  mask  patterns,  the  point  spread  function,  and  possibly  other  mathematical  models  of 
other  optical  components  in  the  system.  Recall  from  the  description  of  the  systems  in  Section  2  that  the  mask 
patterns  are  used  to  implement  the  spatial-temporal  varying  integration  aperture  of  the  detector.  The  unknown 
pixel  values  are  obtained  by  solving  Equation  41.  In  the  presence  of  noise  however,  only  an  approximate  solution, 
p,  can  be  obtained  because  the  noiseless  measurement  is  not  available.  Also,  the  problem  becomes  ill-conditioned. 
One  method  for  finding  the  approximate  solution  is  the  regularized  least  squares5 


P  = 


n  -l 
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Mrm  =  R(ni)m 


(42) 


where  pi  are  regularization  parameters,  H \  are  linear  operators  on  the  solution,  and  the  superscript  T  denotes 
the  matrix  transpose.  Borrowing  notation  from  Bertero,5  the  resulting  matrix  multiplication  and  additions  are 
all  grouped  together  into  R (pi)  for  notational  ease.  In  the  absence  of  any  regularization  R (pi  =  0)  =  M-"1. 
The  linear  operators  Ht-  are  modeled  after  a  priori  knowledge  on  the  solution.  For  example,  if  the  minimal  norm 
solution  is  desired  the  linear  operator  is  the  identity  matrix;  if  the  solution  is  smooth  the  linear  operator  could 
be  the  Laplacian  so  that  the  sharp  transitions  are  minimized.  When  constructing  the  linear  operators  one  must 
keep  in  mind  the  arrangement  of  the  pixel  values  in  the  vector  p. 

From  Equations  41  and  42  we  can  write  the  approximate  solution  in  terms  of  a  noiseless  approximation  term 
and  a  propagated  noise  term 

p  =  R  (pi)m  +  R(pi)r)  (43) 

In  the  absence  of  noise  and  regularization  the  approximate  solution  reaches  the  exact  solution 
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The  exact  solution  is  defined  by  Equation  44  and  should  be  interpreted  as  the  “model  limited”  reconstruction  of 
the  image  plane,  not  the  object  plane.  The  term  “model  limited”  refers  to  the  fidelity  of  the  measurement  model 
represented  by  the  measurement  matrix  M.  With  the  approximate  and  exact  reconstructions  defined,  we  can 
express  the  reconstruction  error  e  as 


£  =  P-P 

e  =  -  p)  +  (45) 

The  relative  reconstruction  error  is  a  more  useful  quantity  and  it  is  given  as 


RRE  = 


M 

IN 


(46) 


For  the  regularized  least  squares  solution,  the  relative  reconstruction  error  depends  on  the  detector  noise,  p,  and 
on  the  regularization  parameters,  pi.  If  the  image  plane  is  illuminated  by  a  constant  intensity  the  value  of  the 
relative  reconstruction  error  can  be  related  to  the  per  pixel  signal  to  noise  ratio  ( SNRp ).  The  relationship  is 
given  by 


SNRp  = 
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(RRE)2 


(47) 


The  reconstruction  error  of  Equation  45  is  composed  of  two  terms,  the  approximation  error  ( R(jui)m  —  p) 
and  the  noise  propagation  (R (pi)p).  Expanding  R (/*$)  as  in  Equation  42,  one  observes  that  the  approximation 
error  is  directly  related  to  the  regularization  parameters  and  the  noise  propagation  term  is  inversely  related  to 
the  regularization  parameters.  When  the  measurement  matrix  is  only  mildly  ill-conditioned  and  the  regular¬ 
ization  parameters  are  small,  the  noise  propagation  term  dominates  the  reconstruction  error.5  For  small  pixel 
count  images  (64x64)  high  regularization  is  not  desired  as  it  leads  to  blur  spanning  several  pixels.  Therefore, 
practical  measurement  matrices  have  to  require  only  mild  regularization.  In  this  case  the  reconstruction  error  is 
-approximated  by 

£  =  R(jm)v  (48.) 


The  noise  in  Equation  48  is  the  detector  noise  considered  in  Section  3.  We  define  the  reconstruction  noise  gain 
as 
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(49) 


As  defined,  the  reconstruction  noise  gain  is  independent  of  the  noise  level  and  can  be  computed  simply  by 
generating  all  white  Gaussian  noise  of  any  variance.  If  the  pixel  count  is  not  large  however,  the  average  of  several 
noise  realizations  might  be  necessary. 

In  a  similar  manner  to  the  reconstruction  noise  gain  we  define  the  reconstruction  signal  gain 


_  ||R(/^H|2 
Gs  — — 


(50) 


where  m  is  the  noiseless  signal.  Unlike  G ^  the  reconstruction  signal  gain  is  dependent  on  the  signal  because  R(pi) 
tends  to  filter  the  spatial  frequency  components  of  the  signal  by  different  amounts.  Also,  if  the  encoding  mask 
is  not  balanced,  R (pi)  might  decode  the  measurements  with  different  gains.  An  example  of  balanced  encoding 
is  the  Hadamard  mask.6  To  circumvent  this  we  propose  using  a  gain  range  rather  than  a  single  gain  value.  One 
limit  of  the  gain  is  calculated  for  a  uniform  image  p  (low  spatial  frequency,  low  sparsity)  and  the  other  for  a  one 
pixel  wide  impulse  image  (high  frequency  limit,  high  sparsity).  Using  the  noise  gain  and  the  signal  gain  range 
we  can  relate  the  detector  noise  ( SNRad )  to  the  per  pixel  SNR  range  (SNRpp) 

=  G^_ 

SNRpr  GSr  1  ’ 

The  gain  ratio  approximates  the  ratio  of  SNRs  defined  by  Equations  24  and  47  as  long  as  the  detector  noise  is 
the  main  contributor  of  the  reconstruction  error. 


For  the  64x64  pixel  conjugate  plane-point  system  that  we  have  implemented  and  demonstrated,7'8  the  SNR 
gain  ratio  ranges  from  2  x  106  for  the  low  sparsity  image,  to  2  x  103  for  the  high  sparsity  image.  For  the  32x32 
pixel  version  of  the  same  imager  the  SNR  gain  ratio  ranges  from  6  x  104  to  3  x  102.  For  the  64x1  line  imager 
version  of  the  device  the  SNR  gain  ranges  between  440  and  20. 

The  32x32  pixel  imager  performs  better  than  the  64x64,  and  the  line  imager  performs  significantly  better 
than  the  two  dimensional  imagers.  The  reason  is  simply  the  degree  of  dependence  between  measurements.  The 
measurement  dependence  increases  with  the  number  of  pixels  for  a  given  imager  physical  dimension.  Another 
important  observation  is  that  reconstructions  for  sparse  images  have  significantly  (orders  of  magnitude)  lower 
gain  ratio  than  the  reconstructions  for  non-spaise  images.  For  more  details  on  our  imager  the  reader  can  refer 
to  our  previous  publications.7,8 

Our  imager  can  be  modified  to  implement  a  Simplex  Code  Mask  (SCM).  The  measurement  matrix  that 
results  from  an  SCM  consists  of  l’s  and  0’s.  According  to  Harwit6  SCM  make  the  best  masks  when  entries  0 
and  1  can  be  used.  For  a  64x1  imager  with  SCM  the  gain  ratio  ranges  from  64  to  2  without  any  regularization, 
and  can  be  even  lower  if  regularization  is  used. 

The  analysis  presented  in  this  section  applies  only  to  the  regularized  least  squares  method  of  image  recon¬ 
struction.  Other  reconstruction  methods,  especially  non-linear,  exist  that  provide  better  immunity  to  noise. 
The  gain  ratio  numbers  presented  above  only  pertain  to  the  current  implementation  of  our  imager  and  we  are 
investigating  methods  of  breaking  the  dependence  between  measurements  that  results  in  high  noise  gain. 

8.  CONCLUSIONS 

In  this  article,  a  series  of  imaging  configurations  for  active  sensors  has  been  defined.  The  SNR  for  each  pixel  has 
been  calculated.  In  some  of  the  schemes  this  SNR  differs  from  the  detector  SNR  due  to  the  necessity  of  significant 
computational  reconstruction  of  the  image.  The  noise  associated  with  linear  methods  of  reconstruction  has  been 
included  in  this  analysis.  The  main  conclusion  of  this  research  is  that  non- traditional  imaging  schemes  involving 
coded  apertures  and  computational  reconstruction  may  provide  better  image  quality  if  reconstruction  methods 
that  are  robust  in  the  presence  of  noise  can  be  achieved.  This  is  a  very  active  area  of  research  in  computational 
optics  and  compressive  imaging  and  is  a  focus  of  our  ongoing  research  in  this  area  at  the  University  of  Memphis. 
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ABSTRACT 

In  this  paper  we  demonstrate  the  use  of  compressive  sensing  to  form  ail  image  with  an  image  plane  random 
mask  and  a  single  pixel  sub-millimeter  wave  receiver.  This  type  of  imaging  device  is  a  practical  solution  in 
domains  where  focal  plane  arrays  do  not  exist.  The  imager  consists  of  a  heterodyne  source  and  receiver  pah-, 
image  forming  opties,  a  spatially  selective  mask,  and  data  acquisition  and  post-processing  hardware  and  software. 
The  spatially  selective  mask  modulates  the  signal  measured  by  the  receiver  which  is  then  sampled  by  an  analog 
to  digital  converter  and  is  post-processed  to  reconstruct  the  image.  The  spatially  selective  mask  can  produce 
image  samples  at  full  video  rates.  The  post-processing  used  for  this  research  consists  of  a  sparseness  inducing 
transformation  on  the  measurements  and  application  of  compressive  sensing  reconstruction  algorithms.  We 
show  several  images  acquired  and  reconstructed  using  this  system.  While  the  data  acquisition  of  this  system  is 
real  time,  the  processing  currently  must  be  done  offline.  We  comment  on  the  performance  improvement  using 
compressive  sensing  methods. 
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1.  INTRODUCTION 

Compressive  sensing  (CS)  has  been  an  active  research  area  recentty.  It  exploits  the  low  information  rate  hidden  in 
a  signal.  Most  signals  are  sparse  in  a  specific  domain.  CS  uses  this  hidden  information  to  reconstruct  a  signal  from 
far  less  samples  than  required  by  a  conventional  algoritlnn.  There  are  two  main  constraints  imposed  for  an  exact 
reconstruction  from  under  sampled  data:  the  sparsity  of  the  signal  and  the  incoherence  of  the  measurement  and 
compression  matrices.1  The  number  of  non  zero  coefficients  in  a  signal  dictates  the  sparsity  index.  For  an  exact 
reconstruction  the  number  of  samples  should  be  approximately  four  times  the  sparsity  index1  and  the  sparsity 
basis  should  be  incoherent  with  the  measurement  matrix.  In  addition,  the  measurement  matrix  should  follow  the 
Restricted  Isometric  Property  (RIP)  to  ensure  that  a  reduction  in  the  number  of  measurements  does  not  have 
a  significant  effect  on  the  information  content.  This  implies  that  the  reduction  of  dimensions  of  measurement 
matrix  preserves  the  distances  between  the  non  zero  planes  of  the  sparse  vector.2  If  a  sampling  process  satisfies 
these  conditions,  this  problem  can  be  cast  into  Basis  Pursuit  Denoising  (BPDN)  or  Lasso.  There  are  different 
optimization  algorithms  available  to  solve  BPDN  and  Lasso.  Some  -widely  used  algorithms  are  Dantzig  Selector, 
BPDN  and  greedy  algorithms.3-5  These  algorithms  are  proven  to  converge  and  axe  tractable.6 

If  the  measurement  matrix  follows  the  properties  mentioned  above,  then  we  can  use  CS  methods  for  re¬ 
construction  by  using  fewer  measurements.  To  reconstruct  an  undersampled  signal  using  CS  theory,  we  need 
knowledge  of  the  spaxsifying  basis  for  SMMW  images.  For  this  purpose  we  used  an  online  dictionary  learning 
algoritlnn  to  find  a  sparsity  basis  in  which  the  signal  can  be  represented  by  fewer  atoms  and  is  also  incoherent 
with  the  measurement  matrix. 

In  tliis  work,  we  consider  the  problem  of  estimating  an  undcrsarnpled  sub-millimeter  wave  line  image.  Earlier 
methods  solved  an  inverse  problem  for  signal  reconstruction  which  requires  as  many  or  more  measurements  than 
the  signal  dimension.  We  address  the  issue  of  reconstruction  from  less  measurements  than  required  by  least 
square  or  inverse  solution  and  show  that  the  number  of  measurements  required  for  optimum  reconstruction  is 
less  than  the  dimensions  of  the  signal  for  a  specific  error  bound. 

The  measurements  obtained  are  transformed  by  the  sparsity  matrix  and  along  with  the  product  of  the 
measurement  matrix  and  sparsity  basis  axe  passed  to  the  reconstruction  algorithm.  We  cast  the  problem  as  a 
basis  pursuit  optimization.  The  basis  pursuit  optimization  estimates  a  solution  to  a  set  of  uuderdetermined  linear 
Equations.  If  the  objective  function  used  is  the  l\  norm  minimization  and  the  RIP  is  satisfied,  we  are  guaranteed 


to  find  the  sparsest  solution  consistent  with  the  measurements.7  Tliis  problem  falls  into  the  convex  optimization 
category,  ensuring  the  solution  is  unique  and  tractable.  As  mentioned  above  there  are  a  lot  of  libraries  available 
to  solve  convex  optimization  problems.  We  used  disciplined  convex  optimization  CVX  modeling  system  to 
implement  BP  optimization.8  CVX  uses  a  base  library  of  convex  functions  and  sets.  The  solution  obtained  is  an 
estimate  of  the  coefficients  in  the  sparse  basis.  The  actual  signal  is  obtained  by  transforming  the  coefficients  back 
to  the  spatial  domain.  We  demonstrate  this  method  by  reconstructing  images  from  simulations  and  experimental 
measurements. 


2.  THEORETICAL  BACKGROUND 

2.1  Compressive  sensing 

CS  is  a  relatively  new  approach  that  provides  a  mechanism  to  sample  a  signal  below  Nyquist  sampling.2’ 11  CS 
imposes  some  conditions  on  the  characteristics  of  the  signal  under  consideration  and  guarantees  a  reconstruction 
from  fewer  samples  than  required  for  conventional  reconstruction.  Consider  a  signal  x  to  be  reconstructed  from 
the  measurements  v.  We  can  express  the  relationship  as 

y  =  Ax +  cu  (1) 

where  y  is  the  measurement  vector  of  length  ky  x  is  the  unknown  signal  to  be  reconstructed  with  n  dimensions. 
The  matrix  A  is  the  measurement  matrix  and  u  is  Gaussian  noise  (0,  <r).  We  consider  the  under  determined 
problem  where  k<<n.  A  signal  has  to  be  sparse  in  a  particular  domain  to  get  an  exact  reconstruction  using  CS 
methodology,  This  implies  we  should  be  able  to  represent  the  signal  in  that  domain  by  a  linear  combination  of 
only  a  few  atoms/columns  of  a  dictionary.  For  instance  if  x  is  sparse  in  a  specific  domain  <^tlien 


X  =  <f)Z 


(2) 


y  =  A<f>z  +  to  (3) 

where  the  vector  2  is  a  sparse  vector  with  very  few  non  zero  elements.  The  sparsity  index  is  the  number  of 
nonzero  elements  in  the  coefficient  vector  2.  If  S  is  the  sparsity  index  of  a  signal  in  a  particular  dictionary  and 
n  is  the  dimension  of  the  signal  then  the  nmnber  of  measurement  required  to  reconstruct  x  should  obey2’12 

k  >  CSlogfn)  (4) 

where  C  is  a  positive  constant. 2It,  depends  on  the  coherence  measure  of  measurement  matrix  and  the  sparsity 
basis.  The  coherence  measure  is  defined  as  follows.  If  a*  and  t pj  represent  the  columns  of  A  and  <f>  respectively 
then2 


(X  = 


y/n 


maxl<i,j<n  |(C4,^>| 


The  matrices  are  incoherent  if  \x  satisfies  the  inequality 


(5) 


/*  <  1  (fi) 

To  reconstruct  from  fewer  samples,  we  need  to  make  measurements  such  that  they  are  incoherent  until  the 
sparsity  basis  matrix  and  the  product  of  the  measurement  matrix  with  the  sparsity  basis  matrix  should  satisfy 
the  RIP.  The  RIP  as  given  by  C-andes8  is  satisfied  if 

a-6s)Ml<\\AMl<  (i  +  fe)IMI*  (7) 

where  the  product  A<fi  is  the  product  of  the  measurement  matrix  with  the  sparsity  basis  matrix  and  z  is  any 
sparse  vector.  The  finite  constant  6s  is  the  restricted  isometry  constant  and  is  given  by  Candos.8  The  REP 


ensures  that  the  transformation  A<f>  preserves  the  distances  between  the  non  zero  planes  of  sparse  vectors.  Said 
another  way,  the  satisfaction  of  the  RIP  is  equivalent  to  the  requirement  that  the  largest  eigenvalue  of  A<p(A<f>)T 
has  in  the  interval  [1  +  63)  1  -  d,]. 

If  the  incoherence  and  the  RIP  are  satisfied  we  can  use  CS  reconstruction  algorithms  for  solving  tliis  under- 
determined  problem  of  linear  Equations.  As  mentioned  before  different  optimization  techniques  such  as  Dantzig 
selector,  Lasso,  or  BP  can  be  used  for  estimating  a  solution.  We  formulate  the  problem  as  BP  and  optimize  it 
using  Self  Dual  Minimization  (SeDuMi). 14  Accor  ding  to  CS  Theory  if  we  minimize  the  /i  norm,  it  gives  us  the 
sparsest  unique  solution  to  tliis  convex  problem.13 

minimize  ||x||i  subject  toy  =  Ax  (8) 

If  we  decompose  x  into  <f>z  and  minimize  the  l\  norm  subjected  to  the  constraints  below,  then 

minimize  \\z\\i  subject  to  y  =  A<fiz  (9) 

We  show  that  solving  tliis  optimization  problem  requires  lass  measurements  than  the  former  problem  as  it 
decreases  the  sparsity  index  by  including  piior  information  about  the  signal  structure.  The  signal  x  can  be 
recovered  using  Equation  2. 

In  the  case  of  noisy  measurements  we  formulate  the  problem  as  BPDN 

minimize  \\x\\i  subjectto\\y  —  Ax | [2  <£  (10) 

If  x  —  <f>z  then 

minimize  \  \z\\\  subject  to  ||y  —  A<^z||2  <  £  (11) 

where  e  >  c  and  a  is  the  noise  variance.  The  signal  can  be  recovered  by  taking  the  inverse  transform  of 
thresholded  coefficients  from  Equation  2.  In  this  paper  we  take  the  transform  directly. 

2.2  Dictionary  learning 

According  to  CS  theory  the  signal  has  to  be  sparse  in  a  specific  domain  for  exact  reconstruction.  There  are  many 
ways  to  find  a  sparsity  basis  wliich  can  represent  the  signal  as  a  linear  combination  of  a  few  atoms.  In  this  paper 
we  used  an  online  learning  approach  for  sparse  coding.9  The  algorithm  considers  a  finite  training  set  of  signals 
and  optimizes  an  empirical  cost  function 

'  /»(*)  4  £  £*(**.*)  (12) 

In  Equation  12,  <j>  is  the  dictionary  in  Rmxn,  each  column  of  <j>  is  called  an  atom  which  are  the  sparse  basis  vectors, 
X{  are  the  training  signals  in  Rmxk,  and  l  is  the  loss  function  wliich  is  small  when  <t>  represent  x{  using  fewer 
sparse  basis  vectors  or  atoms.  The  number  of  training  signals  k  is  typically  large  and  each  signal  is  represented 
by  only  a  few  columns  of  <f>.  Tliis  approach  is  fast  and  guaranteed  to  converge.9 


3.  METHODOLOGY 


The  sub-millimeter  wave  imager  consists  of  a  heterodyne  source  and  receiver  pair,  image  forming  optics,  a  spa¬ 
tially  selective  mask,  and  data  acquisition  and  post-processing  hardware  and  software.  The  compressive  sensing 
measurements  are  formed  by  scanning  the  image  with  the  spatially  selective  mask.  The  receiver  measures  the 
energy  that  is  transmitted  through  the  spatially  selective  mask  for  each  pattern  of  the  mask.  Those  measure¬ 
ments  are  registered  with  the  mask  patterns  and  a  system  of  Equation  describing  the  measurements  in  terms 
of  the  mask  patterns  is  formulated.  This  system  of  Equations  is  represented  in  matrix  form  by  Equation  1.  A 
conceptual  sketch  of  for  the  spatially  selective  mask  considered  in  this  paper  is  shown  in  Figure  1.  Here  we 
point  out  only  some  parameters  of  interest  for  the  spatially  selective  mask  used.  The  mask  consists  of  a  spinning 
disk  with  holes  along  a  constant  radius.  10The  holes  are  of  the  same  size  and  are  placed  at  random  spacings  as 
shown  in  the  figure. We  treat  the  hole  diameter  and  maximum  spacings  as  design  parameters  and  investigate  the 
optimum  combination  of  the  two. 

A  measurement  matrix  resulting  from  the  spatially  selective  mask  is  shown  in  Figure  2.  The  hole  diameter  and 
spacing  affect  the  matrix  in  the  following  way.  The  thickness  of  diagonals  is  related  to  the  diameter  of  holes  and 
the  distance  between  the  diagonals  is  related  to  the  spacing  between  holes.  We  optimized  the  spacing  and  diameter 
of  holes  over  a  range  of  permissible  values  by  minimizing  the  mean  square  error  calculated  from  the  reconstruction. 
The  optimum  value  of  diameter  and  spacing  was  used  for  further  computations  and  implementation. 


Figure  1 .  Conceptual  diagram  of  spatial  mask  and  receiver 

Sub-millimeter  wave  images  from  this  system  can  be  well  represented  by  a  sum  of  Gaussians  due  to  their 
inherent  blur,  A  linear  combination  from  a  set  of  basis  functions  spanning  the  entire  space  can  be  used  to 
represent  any  signal.  To  build  this  basis  for  the  signals  of  interest  we  acquired  the  image  of  a  point  source  and 
translated  it  over  the  whole  spatial  range.  Then  the  set  of  signals  were  fed  to  a  dictionary  learning  algorithm 
to  obtain  a  transformation.9  The  parameters  for  algorithm  were  chosen  empirically.  The  dictionary  learning 
algorithm  enforced  minimum  lx  norm'  to  obtain  a  sparse  representation.9  We  use  the  learned  dictionary  as  a 
sparcifying  basis.  As  mentioned  earlier  in  order  to  reconstruct  a  signal  from  an  optimal  number  of  measurements 
the  dictionary,  or  sparcifying  matrix  and  the  measurement  matrix  should  be  incoherent.  The  sum  of  columns  of 
the  gram  matrix  given  in  Equation  5  for  both  matrices  was  found  to  be  0.36,  which  is  in  range  of  Equation  6. 
This  implies  the  incoherence  of  these  two  matrices. 


(a)  (b)  (c) 

Figure  2.  a.  30x64  Matrix  M,  b.  A  64  x  64  Dictionary  D,  c.  Product  of  D  and  M 


Our  problem  is  expressed  as  an  underdetermined  system  of  linear  Equations  given  by  Equation  1.  Wc  cast  the 
problem  as  BP  and  use  the  SeDuMi  solver  for  optimization.  SeDuMi  was  chosen  due  to  its  lower  computation 
time  required  and  consistent  performance  compared  to  other  available  algorithms.  For  recon structions  from 
noiseless  measurements  we  solve  by  using  equality  constraints  and  the  problem  is  formulated  by  Equation  9.  In 
the  case  of  reconstructions  from  noisy  measurements  we  first  estimate  the  value  of  an  error  bound  e  and  formulate 
the  problem  as  in  Equation  11.  The  i-alue  of  the  error  bound  is  kept  greater  than  the  noise  variance.  For  a  fixed 
number  of  measurements  we  estimated  optimum  e  which  minimized  the  MSE  of  the  reconstructed  signal.  The 
optimum  value  of  error  bound  for  a  specific  number  of  measurements  was  used  for  noisy  reconstruction. 

4.  SIMULATION  AND  EXPERIMENTAL  RESULTS  AND  DISCUSSION 

Wc  simulated  90  signals  of  length  64x1  and  a  64x64  measurement  matrix.  Ninety  measurement  vectors  of 
size  64x1  were  calculated.  A  64x64  dictionary  was  obtained  by  training  the  64  samples  of  the  point  spread 
functions  spanning  the  whole  space  shown  in  Figure  2b.  The  90  signals  were  then  reconstructed  using  30 
noiseless  measurements  and  a  truncated  measurement  matrix.  The  optimum  hole  size  and  spacing  was  found 
by  minimizing  MSE  of  the  reconstruction  over  a  range  of  permissible  values  of  hole  sizes  and  maximum  spacing. 
Figure  3  shows  the  variation  of  MSE  with  hole  diameter  and  maximum  spacing. 


MSE  v*  Hoia  Di»m«1»r  and  Maximum  Spacing 


Maximum  Spacing 

Figure  3.  MSE  vs  Hole  Diameter  and  Maximum  spacing  in  mm 


We  simulated  noisy  measurements  with  <j  =  0.2  and,  using  the  previous  results  for  the  optimized  hole  size  and 
spacing,  we  minimized  the  MSE  of  the  reconstruction  over  a  range  of  error  bound  e  starting  from  the  minimum 
e  =  a  .  The  error  bound  against  least  MSE  was  found  to  be  1.2  which  is  used  in  further  analysis.  Figure4  shows 
the  MSE  plotted  against  a  range  of  error  bound  for  a  specific  number  of  measurements. 


Fig.  4b  shows  the  mean  square  error  plot  against  number  of  measurements  for  a  fixed  e  =  1.2.  We  also 
compare  the  MSE  with  and  without  the  aid  of  a  dictionary  for  reconstruction.  The  error  using  the  dictionary  is 
lower  than  without  the  dictionary  when  smaller  numbers  of  measurements  have  been  used.  After  the  specified 
number  of  measurements  is  passed  the  error  starts  to  increase.  The  specified  number  of  measurements  in  this 
case,  were  30  which  shows  optimum  reconstruction  can  be  achieved  by  less  than  half  the  number  of  measurements 
required  by  a  conventional  algorithm  for  a  given  error  bound.  Moreover  this  optimum  reconstruction  is  achieved 
earlier  with  the  aid  of  a  dictionary  in  the  reconstruction  process.  Figure  5a  shows  a  simulated  signal.  Figures 
5b  &  5c  shows  the  reconstruction  using  30  and  60  measurements  respectively.  As  we  increase  the  number  of 
measurements  in  the  presence  of  noise,  quality  of  reconstruction  degrades. 

We  gathered  real  data  using  the  imaging  device  for  the  “M”  object  shown  in  Figure  7a  .  The  value  of  the 
optimum  error  bound  e  is  calculated  by  minimizing  MSE  over  a  range  of  e  for  a  specific  number  of  measurements 
as  shown  in  Figure  6a. 
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Figure  6.  Reconstruction,  optimization  and  analysis  of  real  data  a.  MSE  vs  Error  bound  b.  MSE  vs  no.  of  Measurements 
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Figure  7.  Real  Data  191  (64x1) 


(b) 


vectors  a.  Object  M,  b.  Reconstruction  using  m  =  30 


The  mean  square  error  increases  after  exceeding  the  specific  number  of  measurements  as  shown  in  Figure 
6b.  The  error  is  lower  initially  when  the  dictionary  is  used  in  the  reconstruction  process  before  the  specified 
number  of  measurements.  Figure  7b  shows  the  reconstructed  <£M”  from  30  measurements  for  each  vector. The 
reconstruction  from  GO  measurements  produced  no  solution.  The  results  were  consistent  with  simulations. 

5.  CONCLUSION  AND  FUTURE  WORK 

Compressed  sensing  provides  the  capability  of  reconstructing  an  image  using  fewer  measurements  than  would 
otherwise  be  required.  It  does  this  by  pushing  the  compression  and  encoding  processes  into  the  measurement 
process  itself.  This  makes  it  attractive  for  imaging  devices  that  need  to  reproduce  a  lot  of  pixels  with  very  few' 
detectors.  It  should  be  noted  that  SMMW  imagers  do  not  strictly  fall  into  this  category.  While  it  is  true  that 
focal  plane  array  detectors  do  not  exist  in  this  regime  it  is  also  true  that  relatively  few  pixels  are  needed  for  most 
security  applications.  This  is  due  to  the  wavelengths  involved,  the  targets  of  interest  (concealed  weapons),  and 
the  ranges  involved  (a  few  to  a  hundred  meters).  Under  these  conditions,  images  will  never  be  terribly  large. 
However,  there  are  other  advantages  to  using  compressive  sensing  reconstruction. 

The  SMMW  imager  described  in  this  paper  and  other  papers  by  the  authors  currently  makes  the  same 
number  of  measurements  as  the  desired  image  size.  The  image  size  and  the  frame  rate  determine  the  sampling 
rate  or  data  acquisition  time  of  the  imager.  Siuce,  for  a  given  image  size,  fewer  measurements  are  required  by 
compressive  sensing  reconstruction  methods  we  can  use  the  acquisition  time  that  would  have  been  used  for  the 
remaining  pixels  for  further  detector  integration  to  reduce  noise  or  to  increase  the  frame  rate  of  the  imager. 

The  imager  described  here  could  be  improved  by  improving  the  RIP  property  of  the  measurement  matrix. 
One  w'ay  to  do  this  is  to  add  another  degree  of  freedom  to  the  measurements  derice.  By  letting  the  holes  vary  not 
onljr  in  their  placement  in  angle  along  a  constant  radius  but  to  vary  about  that  radius,  measurement  matrices 
with  improved  properties  may  be  obtained  and  produce  better  reconstructions. 

Ou  the  processing  side,  the  quality  of  the  images  can  be  enhanced  by  thresholding  the  coefficients  before 
transforming  them  back  to  the  spatial  domain.  We  are  currently  exploring  methods  to  find  the  optimum  threshold 
level.  Also  there  is  room  to  improve  the  role  of  the  sparsity  basis  so  that  it  is  more  robust  to  noise  in  the  signal 
and  other  variations.  Sparsity  in  tlie  third  dimension  (time)  can  also  be  explored  to  further  reduce  the  number 
of  required  samples. 
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ABSTRACT 

In  the  absence  of  detector  arrays,  a  single  pixel  coupled  with  a  spatially  selective  mask  has  been  shown  to 
be  a  practical  solution  to  imaging  problems  in  the  terahertz  and  sub- millimeter  wave  domains.  In  this  paper 
we  demonstrate  real-time  two-dimensional  imager  for  sub-millimeter  waves  that  is  based  on  a  spatially  selective 
image  plane  mask.  The  imager  consists  of  a  heterodyne  source  and  receiver  pair,  image  forming  optics,  a  spatially 
selective  mask,  data  acquisition  hardware,  and  image  reconstruction  software.  The  optics  form  an  image  onto 
the  spatially  selective  mask  and  linear  measurements  of  the  image  are  made.  The  mask  must  be  designed 
to  ensure  maximum  transmission,  measurement  linearity,  and  measurement  to  measurement  independence  and 
our  design  parameters  are  presented.  Once  enough  linearly  independent  measurements  are  made,  the  image 
is  reconstructed  by  solving  a  system  of  linear  equations  that  is  generated  from  the  mask  patterns  and  the 
corresponding  measurements.  We  show  that  for  image  sizes  envisioned  for  many  current  applications,  this  image 
reconstruction  technique  is  computationally  efficient  and  can  be  implemented  in  real  time.  We  present  images 
collected  using  this  system,  discuss  the  results,  and  discuss  other  applications  for  some  components  of  the  imager. 

Keywords:  Sub-millimeter  waves,  terahertz,  imaging,  spatial  light  modulator,  spatially  selective  mask,  com¬ 
pressive  sensing,  multi-mode  imaging,  spectrometry 

1.  INTRODUCTION 

Sub-millimeter  wave  imaging  is  a  useful  tool  for  military  and  security  applications.  Sub-millimeter  waves  can 
penetrate  through  many  man-made  materials  and  provide  high  resolution  images  at  stand-off  distances  while 
still  not  suffering  tremendously  from  atmospheric  attenuation.1,2  However,  the  current  state  of  the  art  is  limited 
to  target  space  scanning  systems  that  require  complicated  mechanical  mechanisms  and  hence  tend  to  be  bulky, 
heavy,  and  not  portable.  These  systems  usually  make  use  of  a  single  pixel  detector  that  offers  very  high  signal  to 
noise  levels  due  to  the  maturity  of  the  technology.  Other  imaging  methods,  in  particular  two-dimensional,  that 
perform  well  in  other  modalities  such  as  interferometric  arrays  in  the  millimeter  wave  region3,4  or  focal  plane 
arrays  in  infrared  and  visible  regimes,  have  yet  to  be  demonstrated  in  the  sub- millimeter  wave  regime. 

Awaiting  the  development  of  focal  plane  arrays  for  sub- millimeter  wave  imaging,  other  alternatives  that  ex¬ 
ploit  the  maturity  of  the  single  pixel  detector  need  to  be  explored.  We  have  focused  our  efforts  on  Spatial  Light 
Modulators  (SLM)  for  sub-millimeter  waves.  Spatial  light  modulators  work  in  reflection  mode  or  in  transmission 
mode.  Perhaps  the  most  well  known  reflective  SLM  is  the  Texas  Instruments  Digital  Micro-mirror  Device  (DMD) 
known  for  its  use  in  the  Rice  University  single  pixel  camera,5  or  in  most  modern  day  projectors.  An  example  of 
transmissive  SLM  are  the  Liquid  Crystal  Optical  Modulators  (LOOM)  common  in  display  applications.  The  un¬ 
availability  of  high  power  illumination  sources  in  the  sub- millimeter  wave  regime  and  the  atmospheric  absorption 
suggest  special  consideration  for  the  efficiency  of  SLMs  for  sub- millimeter  waves.  In  this  respect,  practical  im¬ 
plementations  of  optical  systems  for  SLMs,  where  the  image  plane  is  being  modulated,  favor  transmission  mode 
devices.  The  energy  that  passes  through  the  SLM  transmission  mode  device  can  be  collected  and  channeled  to 
the  detector,  whereas  some  of  the  energy  reflected  by  the  SLM  reflection  mode  device  will  be  diffracted  away 
from  the  detector. 
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An  electronically  controlled  SLM  for  sub- millimeter  waves  would  be  ideal  but  unfortunately  one  suitable 
for  imaging  has  not  been  developed  yet.  There  have  been  some  efforts  to  use  metamaterials  to  develop  an 
electronically  controlled  transmissive  SLM6  but  the  modulation  depth  of  that  device  did  not  exceed  40%,  with  a 
minimum  transmission  coefficient  of  0.3  and  maximum  of  0.7.  Higher  modulation  depths  are  preferred  for  most 
sub-millimeter  wave  imaging  applications.  We  have  developed  a  mechanically  scanned  SLM  for  sub- millimeter 
waves.  Although  a  mechanically  scanned  system,  our  device  has  low  mechanical  complexity,  provides  high 
modulation  levels  and  high  scanning  rates,  and  has  low  cost.  At  the  core  of  this  device  is  a  rotating  disk 
with  holes  that  spins  at  a  constant  speed.  The  image  is  formed  on  one  portion  of  the  disk  and  it  is  selectively 
transmitted  through  the  holes  onto  the  receiver.  Linear  measurements  are  made  on  the  image  as  the  disk  rotates. 
An  image  is  reconstructed  using  the  knowledge  of  the  hole  patterns  and  the  measured  signal  from  the  receiver. 
We  call  this  device  a  Spatially  Selective  Mask  (SSM). 

We  have  previously  presented  the  electromagnetic  analysis  and  design  of  this  device  as  well  as  a  working 
example  of  a  line  SSM  as  part  of  a  line  imager.7,8  In  this  paper  we  describe  and  demonstrate  a  two-dimensional 
version  of  the  device  that  is  used  to  form  64x64  pixel  visible  light  and  sub-millimeter  wave  images.  The  visible 
light  images  are  formed  in  transmission  mode  and  the  sub- millimeter  wave  images  are  formed  in  transmission 
mode  and  in  stand-off  reflection  mode.  In  transmission  mode  the  aperture  of  a  object  mask  is  projected  onto  the 
device  and  in  stand-off  mode  the  target  is  placed  10  meters  away  from  the  imager  and  an  image  of  it  is  formed  on 
the  device  through  an  optical  system.  The  paper  is  organized  as  follows:  Section  2  describes  the  device  and  the 
method  of  image  measurement  and  reconstruction.  In  Section  3,  we  describe  the  experimental  setups.  We  show 
and  discuss  the  images  collected  with  each  setup  in  Section  4.  In  Section  5  we  discuss  further  improvements  to 
the  device  and  additional  uses  of  the  SSM.  Section  6  concludes  the  paper. 


Figure  1.  Conceptual  drawing  of  the  spatially  selective  mask  placed  in  the  image  plane  of  the  imaging  system.  Insert: 
arbitrary  division  of  the  imaging  window  into  pixels. 


2.  THE  TWO-DIMENSIONAL  SPATIALLY  SELECTIVE  MASK  AND  THE  IMAGE 

RECONSTRUCTION  METHOD 

A  conceptual  sketch  of  the  SSM  is  shown  in  Figure.  1.  The  device  is  placed  in  the  image  plane  of  the  imager  and 
an  image  is  formed  over  the  imaging  window.  The  image  is  scanned  with  a  spinning  disk  that  is  patterned  with 
holes.  The  detector  is  sampled  continuously  and  each  sample  is  registered  with  the  corresponding  pattern  of 
holes  in  the  imaging  window  at  the  time  of  the  sample.  When  the  required  measurements  are  recorded,  the  image 
is  reconstructed.  The  central  component  of  the  SSM  is  the  spinning  disk  with  holes  located  at  random  positions. 
The  placement  of  the  holes  and  their  sizes  depend  on  the  largest  imaging  wavelength,  Amax.  Because  of  the 
assumptions  of  the  image  reconstruction  method,  the  transmission  of  each  hole  must  be  unity  and  transmission 
from  many  holes  must  be  linear.  Electromagnetic  analysis8  has  shown  that  for  a  disk  of  thickness  around  three 
Amoi  the  minimal  hole  radius  should  be  greater  than  1.4Amax  and  the  minimal  edge  to  edge  separation  between 
holes  should  be  one  Amax.  As  long  as  the  radius  of  the  holes  and  the  depth  increase  proportionally,  transmission 


and  linearity  are  preserved.  Another  component  of  importance  is  the  collecting  horn  placed  behind  the  imaging 
window  and  the  disk.  This  collecting  mechanism  is  necessary  for  coupling  all  of  the  energy  that  makes  its  way 
through  the  holes  into  the  detector.  From  sub-millimeter  wave  frequencies  to  infrared  frequencies  a  horn  is 
preferred  because  of  the  low  loss  and  cost,  but  for  frequencies  in  the  visible  spectrum  a  lens  can  be  used  instead 
to  focus  the  energy  into  the  detector.  An  integrating  sphere  can  also  be  used.  The  imaging  window  is  another 
component.  It  is  an  aperture  that  limits  the  part  of  the  disk  where  the  image  is  formed  and  scanned.  The  size 
of  the  imaging  window  determines  the  physical  size  of  the  image.  The  motor  spins  the  disk  at  a  constant  rate. 
For  standard  video  rate  imaging  (30  frames  per  second)  the  disk  should  spin  at  1800  rotations  per  minute. 

For  reconstruction  purposes,  the  imaging  problem  is  posed  as  a  linear  system  of  equations  and  is  presented 

in  matrix  notation  _ 

M  p  =  m  (1) 

For  an  image  of  nxn  pixels,  the  column  vectors  m,  and  p  have  dimensions  n2  x  1  and  the  matrix  M  is  a  square 
matrix  of  dimensions  n2  x  n2.  The  vector  p  is  composed  of  the  values  of  the  intensity  of  each  image  pixel.  The 
vector  m  =  m'  -hr?  is  composed  of  the  values  of  each  measurement  and  is  called  the  measurement  vector;  m1  is 
the  noiseless  measurement  and  rj  is  the  measurement  noise.  The  matrix  M  is  called  the  measurement  matrix 
and  each  of  its  rows  encodes  the  pattern  of  holes  over  the  imaging  window  for  each  measurement.  Referring  to 
the  insert  in  Figure  1,  each  entry  of  each  row  of  the  measurement  matrix  is  calculated  as  the  fraction  of  the  area 
of  the  corresponding  pixel  covered  by  holes.  For  instance,  =  0.23,  M\% 2  =  0.27  ,  M1>3  =  0,  and  so  on.  The 
entries  of  the  measurement  matrix  can  also  be  interpreted  as  the  contribution  of  each  pixel  to  the  measurement. 
The  sizes  of  the  pixels  are  arbitrary;  therefore  the  imaging  window  can  be  divided  into  as  many  pixels  as  desired. 
However,  as  the  number  of  pixels  increases  the  probability  that  the  rows  of  the  measurement  matrix  remain 
linearly  independent  decreases.  The  linear  independence  of  the  rows  of  the  measurement  matrix  is  assumed  in 
order  to  solve  Equation  1. 

In  constructing  the  measurement  matrix  in  Equation  1  we  make  the  following  assumptions.  Each  measurement 
is  collected  as  the  pattern  of  holes  in  the  imaging  window  is  stationary.  All  the  energy  that  passes  through  the 
holes  is  collected  and  measured  by  the  detector.  The  measured  energy  is  the  sum  of  the  energy  passing  through 
each  hole.  All  of  these  assumptions  are  broken  in  the  actual  implementation.  The  disk  is  rotating  at  a  constant 
speed  and  each  measurement  is  collected  as  the  disk  rotates.  The  majority,  but  not  all,  of  the  energy  that  goes 
through  the  holes  is  collected  and  measured.  The  holes  can  be  considered  as  radiators  by  the  detector  and  the 
measured  energy  is  not  the  same  as  the  sum  of  the  energies  from  each  hole  because  of  interference.9  However, 
in  the  current  implementation  the  error  or  noise  introduced  because  of  these  assumptions  is  overshadowed  by 
the  measurement  noise.  In  Section  5,  we  discuss  how  some  of  the  error  introduced  by  our  assumptions  can  be 
mitigated. 

Equation  1  can  be  solved  simply  by  inverting  the  measurement  matrix  M  and  multiplying  the  result  with 
the  measurement  vector  m;  however,  the  unique  solution  that  is  obtained  is  different  from  the  desired  vector  p 
because  of  the  measurement  noise.  In  addition,  the  measurement  matrix  might  be  ill-conditioned  which  would 
augment  that  difference.  To  obtain  the  desirable  solution  many  inverse  imaging  algorithms10  can  be  used  to  solve 
Equation  1,  as  well  as  compressive  sensing  methods.11  We  use  a  closed  form  regularized  least-squares  method.10 
It  provides  an  instantaneous  solution  and  allows  the  inclusion  of  some  a  priori  information  about  the  solution  in 
the  form  of  regularization  parameters  and  matrix  operators.  Although  iterative  solutions  are  more  flexible  and 
allow  more  a  priori  information  to  be  used  (an  important  example  is  the  non- negativity  constraint)  they  are  slow 
to  produce  solutions  with  regard  to  video-rate  imaging. 

The  regularized  least  squares  solution  is  written  as 


MT  m  (2) 

where  p*  is  the  approximate  solution,  the  superscript  T  denotes  the  matrix  transpose,  the  regularization  parameter 
a  controls  the  degree  of  the  minimization  of  the  norm  of  the  solution,  the  matrix  operators  Hi  are  linear  operators 
on  the  solution  p\  and  the  regularization  parameters  Pi  control  the  degree  of  minimization  of  the  matrix  vector 


MTM  +  aI+Y,PiH?H i 


Figure  2.  Implementation  of  the  SSM,  front  (left),  and  back  (right)  views. 


product  Hi  p7.  The  matrix  operators  H{  are  used  to  enforce  the  a  priori  information  on  the  solution.  For  instance, 
if  we  know  a  priori  that  the  solution  is  smooth,  we  can  minimize  the  first  and  second  derivatives  of  the  solution. 
In  this  case  the  matrices  B7  are  the  Sobel  or  Prewitt  (first  order  derivatives)  and  the  Laplacian  (second  order 
derivative)  operators.12  These  are  the  matrix  operators  that  we  have  implemented  in  this  work.  Because  of  the 
system  optics  and  large  wavelength,  the  images  that  are  formed  on  the  SSM  are  smooth.  The  regularization 
parameters  can  be  optimized  and  several  methods  for  doing  this  exist10  however,  in  the  work  presented  here 
the  regularization  parameters  were  chosen  ad-hoc  to  produce  a  satisfactory  qualitative  solution.  The  optimal 
regularization  parameters  as  well  as  other  matrix  operators  will  be  investigated  in  the  future. 

The  number  of  pixels  in  the  image  can  be  increased  since  the  pixels  are  non-physical,  however  a  64  x  64 
pixel  image  is  sufficient  for  most  stand-off  applications  of  interest.  Assuming  an  F/l  optical  system  with  one 
meter  aperture  and  a  Rayleigh  wavelength  product  1.22A  =  0.5  mm,  the  angular  resolution  of  the  imager  is 
0.5  mill iradian.  In  object  space  at  100  meters  away  this  translates  to  a  5cm  resolution  spot.  Assuming  one  pixel 
sampling  for  resolution  spot  (two  per  airy  disk)  a  64  x  64  pixel  imager  would  image  an  object  space  area  of 
3.2  x  3.2  meters  square.  This  object  space  area  is  sufficient  to  scan  a  highway  lane,  a  few  human  targets,  the 
trunk  of  a  tree,  or  a  helicopter  landing  site. 

3.  EXPERIMENTAL  SETUPS 

The  spatially  selective  device  was  implemented  using  parts  of  a  5.25  inch  hard  drive.  One  of  the  platters  of  the 
hard  drive  was  used  as  the  spinning  disk.  The  radius  of  the  holes  and  their  minimal  separation  were  designed 
for  operation  at  640GHz.  A  hole  radius  of  1mm  and  a  minimal  separation  of  2  mm  were  chosen  to  guarantee 
unity  transmission,  linearity,  the  structural  integrity  of  the  disk,  and  the  availability  of  standard  tools.  With 
these  parameters  as  restrictions,  a  random  pattern  of  431  holes  was  generated  and  the  holes  were  drilled  on  a 
CNC  mill.  The  case  of  the  hard  drive  was  cut  and  modified  for  access  to  the  disk  and  for  easy  mounting  on 
the  optical  stages.  The  electronics  of  the  motor  driver  were  also  modified  so  that  the  disk  could  be  rotated  at 
3000  rotations  per  minute  (RPM).  An  emitter  diode  and  photo  transistor  pair  were  placed  in  the  proximity  of 
the  disk  to  register  the  measurement  samples  with  the  corresponding  hole  patterns  in  the  imaging  window.  A 
34  mm  x  34  mm  imaging  window  aperture  was  machined  on  two  slabs  of  aluminum  and  they  were  mounted  to 
sandwich  the  disk.  Figure  2  shows  a  picture  of  the  front  and  back  of  the  device  with  all  the  components  described 
above  labeled. 

The  SSM  device  was  used  to  form  images  in  three  different  configurations.  In  the  first  configuration  the  device 
was  paired  with  a  visible  light  detector  to  form  a  visible  light  imager.  Although  this  setup  was  intended  to  help 
debug  of  all  the  issues  related  to  the  device  without  concern  for  the  sub- mi  Hi  meter  wave  image  formation  and 
phenomenology  (invisibility  to  the  bare  eye,  specular  nature  of  the  active  sub-millimeter  wave  imaging,  optical 


alignments,  etc),  it  shows  the  frequency  independence  of  this  device.  A  conceptual  sketch  and  picture  of  the 
setup  for  visible  light  are  shown  in  Figure  3.  This  imager  works  in  transmission  mode.  A  red  laser  source  (400nm 
-  700nm)  was  collimated  and  was  masked  by  object  masks  to  form  images  over  the  imaging  window.  A  lens  was 
placed  behind  the  disk  and  the  imaging  window  to  focus  the  energy  into  an  N-Type  Silicon  PIN  Photodetector. 
The  optical  path  from  the  collecting  lens  to  the  photodetector  was  enclosed  by  a  dark  tube  to  eliminate  stray 
light  reaching  the  detector.  The  detector  signal  was  amplified  and  then  sampled  using  a  16-bit  data  acquisition 
card  from  Measurement  Computing.  For  each  revolution  of  the  disk  5000  samples  were  recorded,  triggered  by 
the  rotation  sensor,  and  then  were  filtered  in  Matlab  using  a  low  pass  filter.  The  disk  was  rotated  at  a  constant 
rate  of  50  rotations  per  second.  We  used  4096  consecutive  measurements  of  the  5000  that  were  recorded  to 
reconstruct  a  64  x  64  pixel  image  using  Equation  2. 


Figure  3.  Visible  light  setup,  sketch  (left)  and  actual  implementation  (right). 


In  the  second  configuration,  the  device  was  paired  with  the  sub-millimeter  wave  receiver  to  form  images  in 
transmission  mode.  A  sketch  and  picture  of  this  setup  are  shown  in  Figure  4.  We  use  a  heterodyne  source  and 
receiver  pair  from  Virgina  Diodes.  The  source  and  receiver  operate  at  640  GHz.  In  this  configuration,  the  640 
GHz  source  is  placed  one  meter  in  front  of  the  device  and  is  flood  illuminating  the  imaging  window.  An  object 
mask  is  placed  right  before  the  imaging  window  and  the  projected  image  is  scanned  by  the  spinning  disk.  The 
640  GHz  receiver  is  placed  behind  the  disk  and  measures  the  energy  passing  through  the  holes.  The  intermediate 
frequency  (IF)  of  the  receiver  was  down  converted  from  4.8  GHz  to  2.2  GHz  and  this  new  IF  was  supplied  to  a 
spectrum  analyzer.  The  spectrum  analyzer  was  used  to  band-pass  filter  and  amplify  the  signal.  A  time  sweep  of 
the  signal  triggered  for  each  rotation  was  generated  and  the  video  output  of  the  spectrum  analyzer  was  sampled 
using  a  16-bit  data  acquisition  card  from  Measurement  Computing.  Here  also  5000  samples  are  recorded  per 
rotation  as  the  disk  rotates  at  50  rotations  per  second  and  then  the  signal  is  low-pass  filtered  in  Matlab  and  4096 
consecutive  samples  are  used  to  reconstruct  a  64  x  64  pixel  image  using  Equation  2. 


Figure  4.  Sub- millimeter  wave  setup  in  transmission  mode,  sketch  (left)  and  actual  implementation  (right). 


In  the  third  configuration,  the  device  was  paired  with  the  sub- millimeter  wave  receiver  to  form  images 
in  stand-off  mode.  A  sketch  and  picture  of  this  setup  are  shown  in  Figure  5.  The  640  GHz  source  is  located 
approximately  two  meters  from  the  target  and  flood  illuminates  it.  The  energy  reflected  by  the  target  is  collected 
by  the  imaging  mirror  (in  the  implementation  the  beam  is  folded  by  two  fiat  mirrors)  10  meters  away.  This  mirror 
is  a  section  of  an  ellipsoid  of  rotation  with  one  focus  at  one  meter  and  the  other  at  10  meters.  The  image  is 
formed  on  the  one  meter  side.  The  system  has  an  effective  diameter  of  0.3048  meters  (12  inch),  effective  focal 
length  of  0.9091  meters,  resulting  in  a  F#  of  2.9826.  The  magnification  of  the  system  is  0.1,  the  depth  of  focus 
for  coherent  radiation  is  0.01062  meters,  and  the  depth  of  field  for  incoherent  radiation  is  1.0162  meters.  The 
airy  disk  diameter  is  3.75  millimeters  for  a  wavelength  of  468.43 /xm,  corresponding  to  the  illumination  frequency 
of  640  GHz.  The  imaging  mirror  forms  an  image  of  the  target  in  the  imaging  window  of  the  SSM.  The  image 
is  then  scanned  by  the  spinning  disk  and  the  energy  that  goes  through  the  holes  is  measured  by  the  640  GHz 
receiver.  The  same  data  collection  setup  as  in  the  second  configuration  is  used  to  sample  the  received  signal  and 
reconstruct  a  64  x  64  image  using  Equation  2. 


Figure  5.  Sub- millimeter  wave  setup  in  stand-off  mode,  sketch  (top)  and  actual  implementation  (bottom) 


4.  RESULTS 

For  all  three  imager  configurations  the  reconstruction  was  performed  using  Matlab  software.  The  Matlab  Data 
Acquisition  Toolbox  was  used  to  acquire  the  data  from  the  data  acquisition  device.  Code  was  written  to  im¬ 
plement  Equation  2  and  to  calculate  the  matrices  therein.  The  disk  is  rotated  at  a  rate  of  50  rotations  per 
second  and  therefore  allows  image  formation  at  50  frames  per  second  (fps).  However,  the  data  acquisition  speed 
of  the  current  implementation  is  only  10  fps  and  when  an  image  is  reconstructed  and  displayed  on  the  screen 
the  image  rate  is  6  fps.  These  slower  frame  rates  are  due  to  the  implementation  in  Matlab.  On  a  laptop  PC 
(processor:  Intel(R)  Core(TM)2  Duo  CPU  T6670  @  2.2GHz),  the  data  reconstruction  step  (  matrix|4094x4096) 
vector[4096xi]  product)  in  the  current  Matlab  implementation  takes  only  30  milliseconds  and  the  rest  of  the  loop 


Figure  6.  Targets  for  the  imager  configurations;  (a)  target  mask  used  for  the  visible  light  and  sub-millimeter  wave  trans¬ 
mission  mode  configurations,  (b)  target  for  the  sub- millimeter  wave  stand-off  mode  configuration. 


(a)  (b)  (c) 

Figure  7.  Image  reconstructions  from  each  configuration;  (a)  reconstruction  from  the  visible  light  configuration,  (b)  re¬ 
construction  from  the  sub- millimeter  wave  transmission  mode  configuration,  (c)  reconstruction  from  the  sub-millimeter 
wave  stand-off  mode  configuration. 

time  is  occupied  by  the  data  acquisition,  image  reshaping,  filtering,  and  image  display.  All  these  steps  can  be 
made  faster  using  a  compiled  implementation  and  dedicated  processing. 

For  the  transmission  mode  configurations  we  used  the  same  mask  for  both  visible  and  sub-millimeter  wave 
imagers;  the  mask  is  shown  in  Figure  6a.  An  aperture  in  the  shape  of  a  hand  gun  was  cut  on  an  aluminum  sheet 
and  the  sheet  was  placed  in  front  of  the  SSM  as  shown  in  Figures  3  and  4.  For  the  stand-off  configuration  an 
aluminum  plate  was  made  rough  and  a  piece  of  carpet  with  an  aperture  in  the  shape  of  a  hand  gun  was  placed 
over  it.  This  target  is  shown  in  Figure  6b.  The  target  was  placed  10  meters  from  the  image  forming  mirror,  as 
shown  in  Figure  5,  and  an  image  was  formed  on  the  SSM. 

The  results  of  the  reconstruction  are  shown  in  Figure  7.  In  all  three  cases  the  shape  of  the  gun  can  be 
distinguished  clearly.  The  image  quality  for  the  transmission  mode  configurations  is  better  than  for  the  stand-off 
configuration.  One  of  the  reasons  for  this  difference  is  the  optical  system.  Another  reason  is  the  specular  nature 
of  active  imaging.  Both  contribute  to  the  non-uniformity  of  the  image  intensity  on  the  image  plane.  We  note 
that  in  order  to  acquire  the  image  in  stand-off  mode  (Figure  7c)  ,  the  object  must  be  positioned  precisely.  A 
slight  angular  movement,  with  respect  to  the  optical  axis,  causes  one  part  of  the  object  to  be  imaged  with  higher 
intensity  than  the  other  parts  and  only  it  can  be  imaged. 

Of  the  three  imager  configurations  the  only  configuration  where  all  the  radiation  was  collected  by  the  detector 
was  the  visible  one.  Because  of  this  feature  we  can  reconstruct  images  regardless  of  where  on  the  imaging  window 
they  are  formed.  The  sub-millimeter  wave  configurations  were  missing  the  energy  collection  mechanism  therefore, 


only  the  central  part  of  the  imaging  window  could  be  used  to  form  images.  In  order  to  use  the  extent  of  the 
imaging  window  an  energy  collecting  mechanism  similar  to  the  collecting  lens  in  Figure  3  is  necessary. 

All  three  reconstructions  in  Figure  7  contain  reconstruction  noise.  This  noise  is  an  effect  of  the  measurement 
noise,  the  non-linearity  of  the  measurement,  and  the  assumption  of  static  hole  patterns  during  the  detector 
integration  time.  Of  these,  the  measurement  noise  dominates.  The  reconstruction  noise  appears  in  the  form  of 
low  spatial  frequency  noise  because  the  signal  m,  that  contains  the  measurement  values,  is  low  pass  filtered  prior 
to  obtaining  the  solution  using  Equation  2.  In  addition,  the  measurement  matrix  M  is  ill-conditioned  therefore 
the  regularization  parameters  are  large.  Large  regularization  parameters  result  in  low  pass  filtering  as  well.  As  it 
was  mentioned  previously,  the  regularization  parameters  that  were  used  were  not  the  optimal  parameters.  The 
optimal  reconstruction  parameters  as  well  as  additional  regularization  operators  will  be  investigated  in  the  near 
future. 


5.  FURTHER  IMPROVEMENTS  AND  USES 

The  results  from  all  three  imager  configurations  provide  insights  into  the  future  improvements  needed  for  a 
practical  stand-off  imager.  They  also  suggest  further  uses  of  the  SSM.  In  this  section  we  discuss  the  required 
improvements  further  uses  of  the  device. 

When  the  device  is  used  in  the  visible  imager  configuration  the  whole  imaging  window  can  be  utilized  to 
form  images,  as  contrasted  to  the  sub-millimeter  wave  configurations  where  only  the  central  part  of  the  window 
can  be  used.  The  difference  is  the  energy  collection  mechanism.  Being  able  to  collect  all  the  energy  that  passes 
through  the  holes  is  a  must  for  the  full  utilization  of  the  SSM.  In  addition  all  parts  of  the  imaging  window  should 
be  coupled  to  the  detector  equally.  We  will  investigate  mechanisms  that  can  be  used  to  achieve  this  in  the  near 
future. 

In  this  implementation  of  the  spatially  selective  disk  a  random  pattern  of  holes  was  generated  and  the  holes 
where  drilled  on  the  disk.  The  only  constraints  that  were  enforced  to  generate  the  random  pattern  of  holes  were 
the  minimal  hole  spacing  and  hole  diameter.  There  are  additional  constraints  that  should  be  imposed  on  the 
generation  of  the  hole  pattern:  (1)  the  hole  pattern  should  result  in  a  measurement  matrix  with  low  condition 
number,  (2)  the  hole  pattern  should  maintain  a  uniform  hole  area  over  the  imaging  window  as  the  disk  spins,  and 
(3)  the  hole  area  over  the  imaging  window  should  be  maximized.  The  first  constraint  improves  the  robustness 
to  measurement  noise.  The  second  constraint  improves  the  uniformity  of  the  image  sampling  and  allows  full 
utilization  of  the  imaging  window.  The  third  constraint  increases  the  amount  of  energy  that  is  measured  by  the 
detector  thereby  increasing  the  signal  to  noise  ratio  of  the  measurement. 

Although  non-random  patterns  of  holes  that  satisfy  the  three  additional  constraints  mentioned  above  might 
exist,  random  patterns  are  preferred.  Random  patterns  improve  the  linearity  of  the  device.  To  the  receiver  each 
hole  appears  as  a  radiating  source  and  the  pattern  of  holes  can  be  considered  to  be  an  antenna  array.13  Due  to 
this  antenna  array  an  interference  effect  as  a  function  of  hole  separation  is  observed  in  the  total  detected  energy. 
This  interference  is  minimized  when  the  hole  separations  are  random. 

The  measurement  matrix  M  is  currently  constructed  by  making  the  assumption  that  the  hole  pattern  is 
stationary  during  the  measurement.  This  assumption  is  clearly  incorrect  as  we  are  spinning  the  disk  continuously. 
A  back  of  the  envelope  calculation  shows  that  the  error  introduced  by  the  inaccurate  assumption  is  at  most  less 
than  1%  for  pixels  at  the  edge  of  the  disk,  when  the  disk  spins  at  50  fps,  the  hole  diameter  is  2  mm,  the  disk 
diameter  is  130  mm,  and  the  integration  time  of  the  measurement  is  1  fis.  For  pixels  located  toward  the  center  of 
the  disk  the  error  decreases  as  a  ratio  of  the  circumference  of  the  disk  at  that  position  with  the  circumference  at 
the  edge  of  the  disk.  This  error  will  be  mitigated  in  the  future  by  including  the  non-stationary  assumption  in  the 
calculation  of  the  measurement  matrix.  This  error  can  also  be  decreased  by  slowing  down  the  disk,  decreasing 
the  hole  diameter,  or  by  decreasing  the  integration  time. 

Another  very  important  component  of  the  imager  is  the  optical  system.  The  optical  system  that  is  currently 
being  used  was  designed  for  conjugate  point  imaging  and  was  not  intended  for  wide  area  imaging.  The  current 
system  suffers  from  several  aberrations  and  is  also  an  off  axis  system.  These  factors  contribute  to  a  non-uniform 
intensity  distribution  in  the  image  and  therefore  parts  of  the  image  are  less  bright  then  others  although  they 


might  be  on  the  same  object  plane.  We  are  currently  designing  a  symmetrical  multi-surface  optical  system  with 
smaller  aberrations. 

The  SSM  was  designed  for  operation  at  640  GHz  and  it  was  also  used  to  form  images  in  the  visible  part 
of  the  spectrum.  This  suggests  that  the  SSM  can  be  used  in  other  parts  of  the  spectrum  as  well.  Using  this 
device  and  reflective  optics  we  can  build  a  multi-band  imager  that  provides  optically  registered  images.  The 
frequency  independence  of  the  device  also  suggests  the  use  of  the  device  to  build  a  spectrometer.  The  SSM  can 
be  used  to  scan  a  spectrum  of  the  scene  mapped  spatially  across  the  mask  using  diffracting  optical  elements.  This 
spectrometer  is  similar  to  a  Hadamard  spectrometer14,15  the  only  difference  being  the  scanning  mask  (pattern 
on  the  disk).  Also,  the  SSM  and  optics  can  be  configured  such  that  an  imager  and  spectrometer  utilize  the  same 
SSM  and  optical  system.  This  device  would  produce  images  of  the  scene  as  well  as  the  spectrum  of  the  scene 
but  not  a  spectral  image.  The  spectrometer  devices  require  a  wide  band  receiver.  These  devices  are  currently 
being  investigated. 


6.  CONCLUSIONS 

We  have  built  a  spatially  selective  mask  device  for  image  space  scanning  with  high  modulation  levels,  high 
scanning  rates,  low  mechanical  complexity,  and  low  cost.  The  design  parameters  that  were  identified  in  our 
earlier  publications7-9  as  well  as  some  reconstruction  methods  presented  there  were  used  to  build  the  device  and 
reconstruct  64  x  64  pixel  images.  We  introduced  here  an  additional  method  for  reconstruction.  Through  a  simple 
calculation  we  concluded  that  a  64  x  64  pixel  image  is  sufficient  for  most  applications  of  interest.  We  demonstrated 
the  use  of  this  device  in  three  different  configurations  and  presented  images  collected  and  reconstructed  for  each 
configuration.  The  imagers  as  configured  can  scan  images  at  50  fps;  they  can  record  images  at  10  fps;  they 
can  scan,  reconstruct,  and  display  images  at  6  fps.  The  data  acquisition  and  image  reconstruction  chains  are 
currently  implemented  on  a  personal  computer  using  Matlab  interpreted  code.  The  processes  of  data  acquisition, 
image  reconstruction,  and  image  display  can  be  accelerated  to  produce  images  at  a  rate  of  50  fps  if  the  code  is 
compiled  and  a  dedicated  computer  is  used.  We  also  discussed  further  improvements  necessary  for  a  practical 
stand-off  video  rate  imager  and  also  suggested  further  uses  of  the  spatially  selective  mask  device.  Based  on  the 
results  shown  in  this  paper  we  suggested  that  the  device  can  be  used  as  part  of  a  multi-mode  imager,  as  part  of 
a  spectrometer,  and  as  part  of  an  imager  spectrometer. 
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Abstract:  We  present  a  spatially  selective  mask  that  is  used  with  a  single  pixel  detector  to 
reconstruct  images  in  real-time.  Reconstructed  image  sizes  are  variable;  the  mask  works  in 
multiple  electromagnetic  regimes.  Experimental  results  are  shown. 

OCIS  codes:  (110.3010)  Image  reconstruction  techniques;  (110.6795)  Terahertz  imaging;  (110  1758)  Computational 
imaging 

1.  Introduction 

Single  pixel  imagers  fmd  utility  m  the  parts  of  the  electromagnetic  spectrum  where  focal  plane  arrays  are 
nonexistent  or  are  expensive  to  fabricate,  such  as  in  the  Terahertz  band  (300GHz  -  3THz).  One  method  used  to  form 
images  with  a  single  detector  is  to  scan  either  the  detector  over  the  image  or  the  image  across  the  detector  using 
moving  optics  [1].  Real-time  or  video  rate,  imagers  utilizing  these  methods  are  possible  at  high  frequencies  but  as 
the  frequencies  decrease  the  optical  components  and  the  detectors  become  large  and  more  difficult  to  move. 

An  alternative  imaging  configuration  is  image  space  scanning.  Examples  of  this  configuration  can  be  traced  back 
to  the  early  days  of  television  with  the  Nipkow  disk  [2].  The  Reticle  systems  work  in  a  similar  way  [3].  Another 
example  is  the  spatial  light  modulators  for  millimeter  waves  developed  by  Jacobs  in  the  1960s  [4].  The  Jacobs 
devices  were  frequency  dependent  and  also  were  not  developed  beyond  laboratory  demonstrations.  One  modem 
device  is  the  digital  multi-mirror  device  (DMD)  developed  by  Texas  Instruments.  The  DMD  was  used  successfully 
to  implement  a  compressive  sensing  imager  at  visible  light  frequencies  [5].  However,  because  of  its  size  and  limiting 
physical  capabilities  on  its  driving  mechanisms  the  DMD  cannot  be  used  at  frequencies  such  as  terahertz. 

We  have  developed  a  spatially  selective  mask  for  terahertz  that  can  also  be  used  for  higher  frequencies.  The 
device  is  implemented  using  a  spinning  disk  with  randomly  placed  holes.  The  mask  is  placed  in  the  image  plane  of 
the  imager  and  the  detector  is  placed  behind  the  mask  to  collect  all  the  energy  that  goes  through  the  holes.  One 
image  is  formed  for  each  revolution  of  the  disk.  The  reconstruction  method  allows  for  variable  image  pixel  sizes  and 
performs  at  video  rates.  The  device  can  also  be  used  to  implement  compressive  sensing. 

The  device  and  the  image  reconstruction  methodology  have  been  described  in  detail  previously  [6],  and  are 
briefly  described  next  for  completeness.  We  present  a  laboratory  prototype  of  the  device  that  was  designed  for 
terahertz  frequencies.  While  we  wait  for  the  construction  of  some  additional  components  needed  to  build  a  terahertz 
imager,  we  paired  the  device  with  a  visible  light  detector  for  proof  of  concept  and  to  verify  the  frequency 
independence  property  of  this  device.  We  were  able  to  reconstruct  simple  images  of  32x32  and  64x64  pixels.  Those 
images  are  presented  and  the  results  are  discussed. 

2.  The  spatially  selective  mask  and  image  reconstruction  method 

A  conceptual  sketch  of  the  spatially  selective  mask  is  shown  in  Fig.  1 .  The  mask  is  placed  in  the  image  plane  of  the 
imager  and  an  image  is  formed  over  the  imaging  window.  The  image  is  scanned  with  a  spinning  disk  that  is 
patterned  with  holes.  The  detector  is  sampled  continuously  and  each  sample  is  registered  with  the  corresponding 
pattern  of  holes  in  the  imaging  window  at  the  time  of  the  sample.  Once  enough  measurements  are  recorded,  the 
image  is  reconstructed. 

The  central  component  of  the  mask  is  the  spinning  disk  with  holes  located  at  random  positions.  The  placement  of 
the  holes  and  their  sizes  depend  on  the  largest  imaging  wavelength,  A^^.  Because  of  the  assumptions  of  the  image 
reconstruction  method,  the  transmission  of  each  hole  must  be  unity  and  transmission  from  many  holes  must  be 
linear.  Electromagnetic  analysis  [6],  has  shown  that  for  a  disk  of  thickness  around  three  X™ax  the  minimal  hole  radius 
should  be  greater  than  1 .4  A^  and  the  minimal  edge  to  edge  separation  between  holes  should  be  one  A™ax  As  long 
as  the  radius  of  the  holes  and  the  depth  increase  proportionally,  transmission  and  linearity  are  preserved. 

Another  component  of  importance  is  the  collecting  horn  placed  behind  the  imaging  window  and  the  disk.  This 
collecting  mechanism  is  necessary  for  the  coupling  all  of  the  energy  that  makes  its  way  through  the  holes  into  the 
detector.  From  terahertz  to  infrared  frequencies  a  horn  is  preferred  because  of  the  low  loss  and  cost,  but  for 
frequencies  in  the  visible  spectrum  a  lens  can  be  used  instead  to  focus  the  energy  into  the  detector.  The  imaging 
window  is  another  component.  It  is  an  aperture  that  limits  the  part  of  the  disk  where  the  image  is  formed  and 


scanned.  The  size  of  the  imaging  window  determines  the  physical  size  of  the  image.  The  motor  spins  the  disk  at  a 
constant  rate.  For  standard  video  rate  imaging  (30  frames  per  second)  the  disk  should  spin  at  1 800  rotations  per 
minute. 


Fig.  1  Conceptual  drawing  of  the  spatially  selective  mask  placed  in  the  image  plane  of  the  imaging  system.  Insert:  arbitrary  division  of  the 

imaging  window  into  pixels. 

For  reconstruction  purposes,  the  imaging  problem  is  posed  as  a  linear  system  of  equations  as  shown  in  Eq.  1. 

m  =  M  p  (1) 

For  an  image  of  n  by  n  pixels,  the  column  vectors  m  and  p  have  dimensions  n2  by  1  and  the  matrix  M  is  a  square 
matrix  of  dimensions  n2  by  n2.  The  vector  p  is  composed  of  the  values  of  each  image  pixel.  The  vector  m  is 
composed  of  the  values  of  each  measurement  and  is  called  the  measurement  vector.  The  matrix  M  is  called  the 
measurement  matrix  and  each  of  its  rows  encodes  the  pattern  of  holes  over  the  imaging  window  for  each 
measurement.  Referring  to  the  insert  in  Fig.  1,  each  entry  of  each  row  of  the  measurement  matrix  is  calculated  as  the 
fraction  of  the  area  of  the  corresponding  pixel  covered  by  holes.  For  instance,  M^i  =  0.23,  M|,2  =  0.27,  MJi3  =  0,  and 
so  on.  The  entries  of  the  measurement  matrix  can  also  be  interpreted  as  the  contribution  of  each  pixel  to  the 
measurement. 

The  sizes  of  the  pixels  are  arbitrary;  therefore  the  imaging  window  can  be  divided  into  as  many  pixels  as  desired. 
However,  as  the  number  of  pixels  increases  the  probability  that  the  rows  of  the  measurement  matrix  remain  linearly 
independent  decreases.  The  linear  independence  of  the  rows  of  the  measurement  matrix  is  assumed  in  order  to  solve 
Eq.  1 .  For  a  more  in-depth  discussion  of  the  image  reconstruction  methodology  refer  to  reference  [6]. 

3.  Laboratory  implementation  of  the  device  and  results 

The  spatially  selective  device  was  implemented  using  parts  of  a  5.25  inch  hard  drive.  One  of  the  platters  of  the  hard 
drive  was  used  as  the  spinning  disk.  The  radius  of  the  holes  and  their  minimal  separation  were  designed  for 
operation  at  640GHz.  A  hole  radius  of  1mm  and  a  minimal  separation  of  2mm  were  chosen  to  guarantee  unity 
transmission,  linearity,  the  structural  integrity  of  the  disk,  and  the  availability  of  standard  tools.  With  these 
parameters  as  restrictions,  a  random  pattern  of  431  holes  was  generated  and  the  holes  were  drilled  on  a  CNC  mill. 

The  case  of  the  hard  drive  was  cut  and  modified  for  access  to  the  disk  and  for  easy  mounting  on  the  optical 
stages.  The  electronics  of  the  motor  driver  were  also  modified  so  that  the  disk  could  be  rotated  at  3000  rotations  per 
minute.  An  emitter  diode  and  phototransistor  pair  were  placed  in  the  proximity  of  the  disk  to  register  the 
measurement  samples  with  the  corresponding  hole  patterns  in  the  imaging  window.  The  imaging  window  aperture 
was  machined  on  two  slabs  of  aluminum  and  they  were  mounted  to  sandwich  the  disk. 

Although  the  device  was  designed  for  operation  at  terahertz  frequencies,  we  paired  it  with  a  visible  light  detector 
to  form  a  visible  light  imager.  This  shows  the  proof  of  concept  and  the  frequency  independence  of  the  device.  A 
picture  of  the  setup  for  visible  light  is  shown  in  Fig.  2.  A  red  laser  source  (400nm  -  700nm)  was  collimated  and  was 
masked  by  object  masks  to  form  images  over  the  imaging  window.  A  lens  was  placed  behind  the  disk  and  the 
imaging  window  to  focus  the  energy  into  anN-Type  Silicon  PIN  Photodetector.  The  optical  path  from  the 
collecting  lens  to  the  photodetector  was  enclosed  by  a  dark  tube  to  eliminate  stray  light  reaching  the  detector.  The 
detector  signal  was  sampled  using  a  16-bit  data  acquisition  card  from  Measurement  Computing. 

Three  object  masks  were  imaged  and  the  reconstructed  images  for  two  pixel  resolutions  are  shown  in  Figs.  3  and 

4.  For  each  revolution  of  the  disk  5000  samples  were  recorded  and  filtered  in  Matlab  using  a  low  pass  filter.  For  the 
32x32  image,  1024  staggered  measurements  were  chosen  and  used  to  solve  Eq.  1  for  the  unknown  pixel  values.  For 
the  64x64  image  4096  consecutive  measurements  were  used.  Equation  1  was  solved  by  regularized  matrix  inversion 
which  effectively  acts  as  a  low  pass  filter 


Fig.  2.  Pictures  of  the  imager  setup  for  visible  light  Left*  collimating  lens,  object  mask,  imaging  window.  Center-the  whole  setup.  Right- 

patterned  disk,  collecting  lens,  tube  enclosure,  detector. 


When  displaying  the  images,  the  negative  values  are  truncated  (dark  spots  in  the  images).  The  negative  values  in 
the  reconstruction  and  the  low  spatial  frequency  noise  are  attributed  to  signal  noise,  filtering  (and  regularization), 
and  the  error  in  the  manufacturing  of  the  disk  which  is  manifested  in  an  incorrect  measurement  matrix.  Minimizing 
noise  and  manufacturing  errors  relaxes  filtering  and  regularization  and  improves  reconstruction. 

As  the  pixel  resolution  increases,  image  reconstruction  improves.  This  is  expected  because  the  fmer  sampling  of 
the  image  decreases  the  pixel-image  ambiguity.  It  should  be  noted  that  the  images  are  acquired,  reconstructed  and 
displayed  at  video  rates.  The  images  can  also  be  reconstructed  using  compressive  sensing  methods  and  a  portion  of 
the  measurements.  This  is  currently  being  investigated. 


Fig.  3.  Image  reconstructions  of  32x32  pixels  for  the  three  object  masks,  the  images  are  normalized;  object  masks  are  shown  In  the  inserts. 


Fig.  4.  Image  reconstructions  of  64x64  pixels  for  the  three  object  masks,  the  images  arc  normalized. 
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ABSTRACT 

In  this  paper,  the  design  and  implementation  of  a  sub-millimeter  line  scanning  imager  using  a  novel  image¬ 
forming  device  is  described.  The  system  consists  of  a  coherent  illuminator,  an  optical  system,  an  image  plane 
mask,  and  a  coherent  detector.  The  image  plane  mask  is  formed  by  making  a  sequence  of  holes  along  a  constant 
radius  of  a  metal  disk.  Spinning  the  disk  scans  the  holes  through  the  image  formed  on  it.  A  detector  placed 
behind  the  spinning  disk  collects  radiation  passing  through  the  holes.  The  holes  are  arranged  in  a  pseudo¬ 
random  pattern.  At  each  detector  sample  time,  energy  from  a  different  pattern  of  holes  is  collected.  A  rigorous 
electromagnetic  analysis  shows  that,  for  a  certain  minimum  size  and  spacing  of  holes  and  certain  disk  thicknesses, 
these  measurements  constitute  a  Unear  measurement  of  the  energy  in  the  image  formed  on  the  disk.  Using 
techniques  reminiscent  of  those  used  in  compressive  sensing,  the  image  is  then  reconstructed  by  applying  an 
inverse  linear  matrix  transform  to  these  measurements.  We  show  how  simulation  can  be  used  to  optimize  the 
design  of  the  disk.  We  demonstrate  a  laboratory  version  of  this  device  and  discuss  future  efforts  to  systematize 
it.  Extensions  to  full  two-dimensional  imaging  are  also  discussed. 

1.  INTRODUCTION 

The  development  cycle  of  imaging  devices  is  a  recurring  one.  First,  a  single  detector  is  developed  that  allows  the 
conversion  of  radiation  to  a  form  that  can  be  digitized  by  a  computer  or  perceived  by  a  human.  Next,  comes  the 
challenge  of  reducing  the  cost  of  the  detector  and  making  it  more  compact  so  that  staring  arrays  can  be  built. 
This  turns  out  to  be  a  difficult  challenge.  Therefore,  in  the  meantime,  scanning  systems  are  developed  that 
facilitate  the  formation  of  images  at  the  expense  of  mechanical  complexity,  system  size  and  weight,  and  image 
formation  rates.  Finally,  the  goal  of  building  staring  arrays  is  achieved  and  scanning  systems  become  obsolete. 
However,  scanning  systems  play  an  important  role  because  they  fill  the  time  gap  between  the  development  of 
the  single  detector  and  the  staring  array.  Some  examples  of  this  cycle  are  visible  light  imagers,  infrared  imagers, 
and  submillimeter  wave  imagers.  To  our  knowledge,  focal  plane  arrays  in  the  submillimeter  wave  regime  have 
not  been  demonstrated  yet,  although  a  method  for  building  compact  linear  heterodyne  receiver  arrays  has  been 
proposed  and  is  in  the  first  stages  of  development.1 

The  complexity  of  developing  scanning  systems  for  sub-millimeter  waves  is  greater  than  it  is  for  the  scanning 
systems  in  other  regimes,  such  as  visible  and  infrared.  First,  the  wavelength  is  larger  therefore  the  size  and 
weight  of  these  systems  are  greater.  Second,  the  radiation  is  not  as  abundant.  Current  scanning  systems  for 
sub- millimeter  waves  are  large  in  size  and  have  image  formation  rates  of  less  than  one  frame  per  second.2  We 
have  developed  a  novel  image  forming  device  that  can  be  packaged  into  a  handheld  box  and  is  capable  of  scanning 
and  forming  images  at  video  rates. 

The  system  consists  of  a  coherent  illuminator,  an  optical  system,  an  image  plane  mask,  and  a  coherent 
detector.  The  image  plane  mask  and  the  image  formation  methodology  associated  with  it  are  the  novel  parts  of 
the  system.  The  image  plane  mask  is  formed  by  making  a  sequence  of  holes  along  a  constant  radius  of  a  metal 
disk.  Spinning  the  disk  scans  the  holes  through  the  image  formed  on  it.  A  detector  placed  behind  the  spinning 
disk  collects  radiation  passing  through  the  holes.  The  holes  are  arranged  in  a  pseudo-random  pattern.  At  each 
detector  sample  time,  energy  from  a  different  pattern  of  holes  is  collected.  These  measurements  constitute  a  linear 
measurement  of  the  energy  in  the  image  formed  on  the  mask.  This  process  can  be  formalized  mathematically 
and  is  expressed  as 
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m  =  Mp  (1) 

where  m  is  a  vector  the  elements  of  which  contain  the  values  of  the  measurements,  M  is  a  full  rank  matrix  that 
encodes  the  pattern  of  the  holes  on  the  disk  for  each  measurement,  and  p  is  the  unknown  vector  that  contains  the 
pixel  intensities.  In  compressive  sensing  terminology,  M  is  the  measurement  matrix  and  m  is  the  measurement 
vector.  The  image  is  reconstructed  by  solving  this  equation  for  p.  Because  M  is  a  full  rank  matrix,  the  solution 
is  easy  to  compute.  This  device  and  the  image  measurement  and  reconstruction  technique  are  described  in  more 
detail  in  a  previous  publication.3  A  conceptual  sketch  of  the  image  plane  mask  and  the  coherent  detector  is 
shown  in  Figure  1. 

The  imager  is  configured  similarly  to  a  system  with  a  focal  plane  array  and  the  image  plane  mask  is  positioned 
where  the  focal  plane  array  would  be.  Because  the  system  is  still  a  scanning  system,  the  available  integration  time 
per  measurement  per  detector  is  much  less  than  the  integration  time  available  to  the  focal  plane  array  detectors. 
This  is  mitigated  in  part  by  the  presence  of  many  scanning  holes  per  measurement  allowing  more  energy  to  go 
through  and  be  measured  by  the  detector.  In  addition,  if  compressive  imaging  techniques  are  used,  the  image 
can  be  reconstructed  by  taking  less  measurements  than  desired  image  pixels.4,5  This  increases  the  integration 
time  available  per  measurement.  The  drawback  of  this  technique  is  the  slow  reconstruction  time  associated  with 
the  iterative  compressive  sensing  algorithms.  This  approach  is  not  employed  in  this  research,  rather  the  number 
of  measurements  is  made  equal  to  the  number  of  desired  pixels  allowing  instantaneous  reconstruction. 

In  this  paper,  we  present  the  results  of  a  three-dimensional  analysis  that  aids  in  the  design  of  the  image 
plane  mask.  The  linearity  of  the  mask  is  investigated.  The  reasons  for  the  non-linearity  of  the  mask  and  a 
method  by  which  this  mask  is  made  nearly  linear  are  discussed.  Following  this  introduction  we  briefly  present 
the  electromagnetic  analysis  methods  used  in  this  paper.  The  results  of  the  analysis  are  discussed  and  the 
design  parameters  are  given.  Next,  we  describe  the  implementation  of  the  system  and  show  experimental  results 
considering  the  results  of  the  electromagnetic  analysis.  We  discuss  our  approach  to  systematizing  the  device 
so  that  the  images  can  be  scanned  and  reconstructed  at  video  rates.  We  also  discuss  the  extension  to  a  two- 
dimensional  imager.  The  paper  concludes  with  a  summary. 

2.  THEORETICAL  METHODS 

To  design  the  image  plane  mask  we  utilize  a  three-dimensional  electromagnetic  analysis.  The  three-dimensional 
structure  of  interest  is  assumed  to  be  a  perfect  electric  conductor  (PEC).  The  field  incident  on  the  structure  is 
an  elliptically  polarized  tapered  Gaussian  beam  given  by  Braunisch.6  The  observables  of  interest  are  the  power 
incident  on  the  structure,  the  power  reflected  from  the  structure,  and  the  power  transmitted  by  the  structure. 
The  powers  are  obtained  once  the  far  incident  and  scattered  fields  are  known.  The  far  incident  field  is  derived 
using  the  stationary  phase  approximation.7  Using  the  incident  field  on  the  structure  we  solve  the  Electric  Field 
Integral  Equation  (EFIE)  for  the  currents  using  the  Method  of  Moments  (MoM)  technique.  The  currents  are 
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Figure  2:  Cross-section  cut  of  the  scattering  geometry. 


used  to  find  the  far  scattered  fields  with  the  help  of  the  far  field  approximation.  Below  we  summarize  the  main 
equations  used  in  the  analysis.  For  the  complete  derivations  of  the  equations  and  the  numerical  techniques  used 
see.8 

To  formalize  the  problem  the  scattering  object  is  assumed  to  be  an  open  scatterer.  The  open  scatterer  can 
be  extended  and  deformed  to  adhere  to  the  scattering  structure  of  interest.  A  cross-section  of  the  scattering 
geometry  is  shown  in  Figure  2.  The  scatterer  is  illuminated  with  the  Gaussian  tapered  beam  from  above  (z  >  0) 

.  The  incident  field  induces  currents  on  the  scatterer  which  radiate.  Therefore,  the  total  field  anywhere  in  space 
is  composed  of  the  incident  field  and  the  scattered  field  radiated  by  the  scatterer 

E(R)  =  E,(K)  +  EM)  (2) 

The  EFIE  for  the  structure  is  obtained  in  the  following  manner.  The  well-known  vector  wave  equation  is 
transformed  into  an  integral  equation  using  the  well-known  vector  form  of  Green’s  theorem  and  the  free  space 
Green  function  dyad  Q ^  given  by9 


g.CS.fi')  -  (i  +  pVV)  (3) 

where  l  =  xx  +  yy  +  ££,  is  the  unit  dyad  (or  idem  factor),  k2  =  oj2e0fjL0  is  the  free  space  wave  number, 
R  =  xx  -J-  yy  +  zz  is  a  vector  in  Cartesian  space  noting  the  observation  point,  Rl  is  a  vector  noting  the  source 
point,  and  V  =  -J-  y ^  -f  is  the  gradient  operator.  The  Green  function  dyad  satisfies  the  wave  and 

radiation  equations.9  Using  this  fact  and  applying  vector  identities,10  we  obtain 

E(R)  =  -j)  x  H{g))  ■  Q^(g,g)  +  (n  x  E(g))  ■  V'  x  £,(£',£)]  ds‘  (4) 

s 

where  E(R)  is  the  electric  field  in  a  volume  V  enclosed  by  the  surface  5,  n  is  the  outward  normal  to  5, 
is  the  magnetic  field  on  the  surface  contour. 


Referring  to  Figure  2,  Equation  4  is  applied  to  the  following  fields  in  the  respective  regions:  the  scattered  field 
in  the  volume  V'+enclosed  by  S+,  Sc,  and  S+ ;  the  incident  field  in  the  volume  ^“enclosed  by  S+,  Scy  and  S^; 
the  total  field  in  the  volume  ^“enclosed  by  5S“,  Sc ,  and  S In  all  three  cases,  the  integral  over  s£/~  vanishes 
because  the  scattered  and  incident  fields  satisfy  the  radiation  condition  in  the  respective  regions.  Combining  and 
adding  the  three  resulting  equations  as  the  contours  S+  and  S~  approach  Ss  and  using  equation  2,  results  in 
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+(n  x  (E+  -  E~))  ■  V  x  gj  ds'  ReV~UV+ 


(5) 


Equation  (5)  gives  the  scattered  field  in  the  volumes  V~  and  V+  in  terms  of  the  fields  on  the  scatterer  surface 
Ss.  The  superscripts  +  and  -  indicate  fields  above  and  below  the  scatterer  and  n  is  the  unit  normal  pointing 
into  V+.  This  equation  is  made  more  specific  by  applying  the  well-known  PEC  boundary  conditions11  of  the 
scatterer 

1  r  e* \Z-R'\ 

=  Re  v-uv\  (6) 

where  is  the  surface  current  on  the  scatterer  and  the  free  space  Green  function  dyad  is  written  explicitly. 
Equation  (6)  is  used  to  find  the  scattered  field  in  the  volume  V  once  the  surface  currents  are  known.  The  final 
form  of  the  EFIE  is  obtained  using  Equation  2  in  Equation  6  and  evaluating  Equation  6  on  the  boundary  of  the 
PEC  scatterer 
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The  unknown  currents  are  obtained  by  solving  Equation  7  using  the  Method  of  Moments. 
The  incident  field  on  the  scatterer  is  given  by8 

2t t  -*/2 

KM)  =  fdtjddk2  sin  6  cos  0  x 

0  0 

exp{ik[xsinOcos<j)  +  ysindsm<j)  —  z  cos 5]}  x 


(7) 


(8) 


This  is  a  Gaussian  beam  with  spatial  center  at  the  origin  of  the  coordinate  system  and  e{9,  <j>)  and  are 

the  polarization  vector  and  Gaussian  spectrum  respectively,  given  in.6 

The  far  incident  field  is  calculated  from  Equation  8  using  the  method  of  stationary  phase  for  double  integrals 
as  presented  in  Mandel  and  Wolf.7  For  z  >  0,  and  0  <  0S  <  7t/2,  0  <  <f>s  <  27t,  the  far  incident  field  is  given  by 
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Similarly,  the  far  incident  field  for  z  <  0,  and  7t/2  <  0S  <  7r,  0  <  <j>s  <  2? r,  is  given  by 
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where 
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is  the  far  incident  field  amplitude  and  it  is  assumed  that  0  <  <f>s  <  2n. 

The  far  field  is  obtained  by  approximating  Equation  (6)  for  the  large  argument  \R  -  R'\  and  making  the  far 
field  approximation.12  The  far  scattered  field  is  given  by 
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where 


Es(0, 4)  =  I  (£(#)  -  ob  ■  J,(g))  e-w e>dS' 

S, 


(12) 

(13) 


is  the  far  scattering  field  amplitude.  The  unit  vector  o  is  in  the  direction  of  the  observation  and  is  given  by 

o  =  xsin  8  cos  <f>  +  j/ sin  #  sin  <j>  +  z  cos#  (14) 

Because  of  the  conservation  of  energy,  the  power  incident  on  the  structure  is  equal  to  the  sum  of  the  powers 
reflected  by  and  transmitted  through  the  structure.  Because  the  structure  is  a  PEC,  there  is  no  absorbed  power. 
Therefore8 

Pi  =  Pt  +  Pr  (15) 

where  Pj  is  the  incident  power  given  by 


Pr  is  the  reflected  power  given  by 


Pt  is  the  transmitted  power  given  by 
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For  the  numerical  solution  of  Equation  7  the  structure  of  interest  is  modeled  using  triangular  patches.  The 
famous  Rao-Wilton-Glisson  (RWG)  basis  functions13  are  used  as  basis  and  testing  functions  for  the  MoM.  The 
resulting  matrix  equation  is  solved  using  available  software  such  as  Matlab.  Once  the  surface  currents  on  the 
structure  are  found,  the  desired  power  quantities  are  calculated  with  the  help  of  the  equations  presented  above. 
For  detailed  explanation  of  the  equations  presented  above  and  the  numerical  methods  associated  with  them  refer 
to.8 


3.  ELECTROMAGNETIC  ANALYSIS  RESULTS  AND  DESIGN  PARAMETERS 

The  analysis  method  presented  above  is  used  to  design  the  parameters  of  the  scanning  holes.  We  are  interested 
in  the  radius,  depth,  and  minimal  proximity  of  the  holes.  From  previous  results  8  the  depth  of  the  holes  does 
not  have  a  noticeable  effect  on  the  transmission  of  the  radiation  through  the  holes  as  long  as  the  radius  of  the 
hole  is  larger  than  the  first  cutoff  radius  of  a  circular  waveguide  and  the  depth  is  comparable  to  the  diameter 
of  the  hole.  For  holes  with  radii  less  than  the  cutoff  radius,  transmission  decreases  exponentially  with  depth. 
As  it  will  be  shown  in  the  results  presented  below,  we  are  only  interested  in  holes  with  large  radii.  Therefore, 
from  a  design  perspective,  the  hole  depth  is  a  free  parameter.  The  thickness  of  the  disks  available  to  us  is 
approximately  2.71 12A,  where  the  wavelength  A  =  468.43/xm,  corresponding  to  a  frequency  of  640GHz.  Because 
of  this  restriction,  the  hole  depth  parameter  is  fixed. 


(a) 
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Figure  4:  Power  transmission  coefficient  as  a  function  of  hole  radius  for  hole  depth  of  2.71 12A. 


The  power  transmission  coefficient  of  the  holes  with  depth  2.71 12A  is  investigated  as  a  function  of  the  hole 
radius.  The  three-dimensional  geometry  of  the  structure  used  in  this  analysis  is  shown  in  Figure  3.  The  geometry 
is  finite  and  has  a  diameter  of  8A.  The  incident  field  has  a  tapering  of  1.5A  (the  tapering  parameters  is  related 
to  the  variance  of  the  Gaussian  taper),  is  normally  incident,  and  is  circularly  polarized.  The  incident  beam 
resembles  a  blur  spot  approximately  5 A  in  diameter.  Because  the  incident  field  is  tapered,  the  structure  can  be 
made  finite  and  the  transmission  of  the  hole  can  be  measured.  The  radius  of  the  hole  is  varied  from  0.2 A  to  2. 05 A 
in  steps  of  0.05A.  The  power  transmission  coefficient  is  calculated  as  the  ratio  of  the  incident  power  on  the  top 
aperture  of  the  hole  to  the  power  transmitted  in  the  lower  hemisphere  (z  <  0).  The  results  are  shown  in  Figure 
4. 

The  transmission  coefficients  for  certain  radii  are  greater  than  unity.  The  reason  for  this  is  that  only  the 
power  incident  on  the  hole  aperture  from  above  is  used  as  the  input  power  reference.  The  fields  radiated  by  the 
edge  currents  at  the  hole  apertures  are  not  included  in  the  input  power  reference  but  they  are  accounted  for  in 
the  far  field  measurement.  Hence,  the  calculated  power  transmission  coefficient  is  greater  than  unity.  The  edge 
current  effects  become  less  evident  as  the  radius  increases.  When  the  radius  is  larger  than  1.4A  these  effects  are 
almost  unnoticeable  and  therefore  the  far  field  measurement  results  only  from  the  power  incident  on  the  hole 
aperture.  In  this  case,  the  holes  are  said  to  scan  the  image  formed  on  the  image  plane  mask.  This  is  the  desired 
effect  because  we  need  the  radiation  associated  with  the  part  of  the  image  formed  on  the  hole  to  pass  and  the 
part  of  the  image  outside  the  hole  to  be  blocked. 

Another  important  parameter  is  the  proximity  between  two  holes.  This  affects  the  linearity  of  the  structure. 
Cross-sections  of  the  geometries  used  to  investigate  the  minimal  proximity  are  shown  in  Figure  5.  The  depth  of 
the  holes  is  2.7112A  and  the  radius  of  the  holes  is  1A.  For  these  hole  parameters  the  transmission  coefficient  is 


Figure  5:  Cross-sections  of  the  geometries  used  to  investigate  the  minimal  proximity  between  holes.  In  the 
three-dimensional  simulation  the  depth  of  the  holes  is  2.7112A  and  the  radius  is  1A.  The  incident  field  is  the 
same  as  in  the  radius  analysis. 
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Figure  6:  Percent  linearity  error  as  a  function  of  the  center  to  center  separation  of  two  identical  holes. 


unity.  For  a  linear  structure,  the  measurement  from  the  structure  in  Figure  5a  should  be  equal  to  the  sum  of 
the  measurements  from  the  structures  of  Figures  5b  and  5c.  Because  the  source  and  the  detector  are  coherent 
the  power  measurement  is  not  expected  to  be  linear. 

To  investigate  the  linearity  of  the  structure  as  a  function  of  the  separation  distance  we  introduce  the  percent 
linearity  error  metric.  The  percent  linearity  error  is  calculated  as  the  difference  of  the  measurement  from  the 
structure  in  Figure  5a  with  the  sum  of  the  measurements  from  the  structures  of  Figures  5b  and  5c.  Then,  the 
result  is  divided  by  the  the  sum  of  the  measurements  from  the  structures  of  Figures  5b  and  5c  and  multiplied 
by  100%.  The  results  are  shown  in  Figure  6. 

Except  when  the  holes  are  very  close  to  each  other  (edge  to  edge  separation<  A)  the  linearity  error  oscillates 
around  zero  with  a  period  of  1A.  This  error  is  predicted  by  the  array  factor  of  this  structure8  because  the 
structure  resembles  two  identical  radiating  antennas  separated  by  a  distance.14  The  error  can  be  eliminated  if 
many  holes  are  placed  at  random  distances  from  each  other.  This  is  advantageous  because  it  not  only  improves 
the  linearity  of  the  structure  but  it  also  does  not  require  any  modification  of  the  scanning  technique.  The  holes 
in  the  line  imager  are  placed  in  random  fashion  as  required  by  the  reconstruction  technique. 

From  the  results  of  the  electromagnetic  analysis,  we  can  conclude  design  parameters  for  the  holes  in  the 
image  plane  mask.  The  radius  of  the  holes  needs  to  be  greater  than  1.4A  but  other  radii  are  also  possible  such 
as  1A.  In  all  these  cases  the  far  field  measurement  results  from  the  field  incident  on  the  hole,  and  not  from  the 
edge  currents.  In  this  case  the  holes  are  effectively  scanning  the  image  formed  on  the  mask.  The  edge  to  edge 
separation  of  the  holes  is  recommended  to  be  greater  than  1A.  This  separation  assures  that  the  only  non-linearity 
is  due  to  the  array  factor  of  the  structure.  The  depth  of  the  holes  was  determined  from  the  available  disk  for  the 


Figure  7:  Grouping  of  pictures  illustrating  the  imaging  system  and  its  parts. 


production  of  the  mask,  however  if  the  depth  parameter  is  to  be  designed  it  should  be  chosen  comparable  with 
the  diameter  of  the  holes  and  such  that  it  does  not  introduce  resonances  (i.e.  edge  currents). 

4.  IMPLEMENTATION  OF  THE  IMAGING  SYSTEM 

A  grouping  of  pictures  illustrating  the  descriptions  presented  in  this  section  is  shown  in  Figure  7.  A  laboratory 
prototype  of  the  image  plane  mask  device  has  been  implemented  using  parts  from  a  5.25  inch  hard  drive.  The 
hard  drive  platters  are  made  of  conducting  materials  and  are  balanced  and  flat.  They  are  used  as  the  disks  on 
which  the  scanning  holes  are  drilled.  The  motor  can  be  driven  at  a  constant  rate  and  the  disk  is  already  mounted 
on  it.  We  have  designed  some  electronics  that  can  drive  the  motor  at  three  revolutions  per  second.  The  motor 
can  rotate  at  speeds  up  to  90  rotations  per  second  using  other  electronics.  The  front  cover  of  the  hard  drive  is 
removed  and  a  window  is  placed  in  front  of  the  scanning  disk.  This  is  the  imaging  window  and  the  optical  system 
forms  the  image  onto  it.  The  extent  of  the  imaging  window  is  22  millimeters  long.  An  opening  is  made  in  the 
back  of  the  case  so  that  the  radiation  is  allowed  to  pass  onto  the  receiver.  An  optical  sensor  is  mounted  in  the 
vicinity  of  disk  edge.  This  sensor  is  used  to  trigger  a  measurement  for  each  rotation  of  the  disk  and  facilitates 
the  registration  of  the  measurements  with  the  position  of  the  disk.  The  receiver  and  source  are  from  Virgina 
Diodes  and  operate  at  640  GHz.  The  signal  is  detected  from  the  receiver  and  is  down-converted  to  4.8  GHz. 
The  down-converted  signal  is  measured  using  a  spectrum  analyzer  (Agilent  SCA  Spectrum  Analyzer  AN1996A). 
The  measurement  data  is  transferred  remotely  to  a  computer  where  it  is  stored  and  later  post-processed  to 
reconstruct  the  image.  For  each  rotation  of  the  disk  1000  data  points  are  collected. 

The  optical  system  consists  of  a  main  elliptical  reflective  surface  with  a  focus  at  one  meter  and  the  other  at 
10  meters.  The  image  is  formed  on  the  one  meter  side.  The  system  has  an  effective  diameter  of  0.3048  meters 
(12  inch),  effective  focal  length  of  0.9091  meters,  resulting  in  a  F#  of  2.9826.  The  magnification  of  the  system  is 
0.1,  the  depth  of  focus  for  coherent  radiation  is  0.01062  meters,  and  the  depth  of  field  for  incoherent  radiation  is 
1.0162  meters.  The  diffraction  spot  diameter  is  3.75  millimeters  for  a  wavelength  of  468.43  fim)  corresponding  to 
the  illumination  frequency  of  640  GHz.  The  system  is  folded  using  two  flat  mirrors.  The  source  is  placed  three 
meters  away  from  the  object  to  flood  illuminate  it. 

Holes  have  been  drilled  on  the  disk  in  a  constant  radius  of  58  millimeters.  This  radius  and  the  extent  of 
the  imaging  window  were  chosen  to  minimize  the  curvature  of  the  scan.  A  single  hole  of  radius  0.5  millimeters 


Figure  8:  Picture  of  the  target. 


(1.06  A)  is  drilled  at  a  distance  from  other  holes  that  have  a  radius  of  1  millimeter  (2.12  A)  and  are  placed  in  the 
vicinity  of  each  other  at  a  random  order  respecting  the  design  parameters  presented  above.  The  single  hole  is 
used  to  perform  a  raster  scan  of  the  image  formed  on  the  imaging  window.  The  radius  of  half  a  millimeter  allows 
high  transmission  (over  98%)  and  high  scan  resolution.  The  disk  is  1.27  (2.7112  A)  millimeters  thick. 

The  imaging  window  is  placed  over  the  disk  so  that  the  image  is  scanned  vertically.  For  demonstration 
purposes,  the  object  is  placed  on  a  translational  stage  and  is  scanned  horizontally.  Vertical  lines  of  the  image 
of  the  object  are  scanned  by  the  image  plane  mask  as  the  object  is  moved  horizontally  and  an  image  of  it  is 
reconstructed  when  the  measured  data  is  post-processed.  The  results  are  shown  in  the  next  section. 

5.  EXPERIMENTAL  RESULTS 

For  this  experiment  we  chose  to  form  an  image  of  the  letter  “M”.  A  picture  of  the  target  mounted  on  the 
translational  stage  is  shown  in  Figure  8.  The  object  letter  is  formed  by  placing  a  piece  of  carpet  with  a  cutout  of 
the  letter  “M”  over  a  plate  of  aluminum.  The  plate  of  aluminum  was  made  rough  to  minimize  specular  reflections. 
The  contour  of  the  letter  “M”  was  approximately  15  to  20  millimeters  wide.  The  height  of  the  letter  was  60 
millimeters.  Because  of  the  characteristics  of  the  optical  system,  the  image  is  expected  to  be  one  tenth  of  the 
object  in  size  and  highly  blurred.  The  target  was  made  small  to  ensure  that  it  was  illuminated  uniformly  by  the 
source.  We  did  not  use  illumination  optics  hence,  the  wavefront  reaching  the  object  was  not  planar  and  images 
of  large  objects  were  not  formed  properly. 

The  results  of  the  raster  scan  (single  hole)  and  linear  measurement  scan  (plurality  of  holes)  are  shown  in 
Figure  9a  and  Figure  9b  respectively.  Referring  to  the  raster  scan  image  which  will  be  used  as  the  reference,  the 
image  is  severely  blurred.  However,  the  structure  of  the  letter  “M”  can  be  identified.  The  blur  is  expected  given 
the  characteristics  of  the  optics.  From  the  measurements  the  image  has  a  height  of  25  samples  corresponding 
to  9.1  millimeters  (the  radius  where  the  holes  are  placed  is  58  millimeters  and  1000  data  points  are  collected 
per  rotation).  This  is  consistent  with  the  calculation  from  the  parameters  of  the  optical  system,  resulting  in  an 
object  size  of  approximately  60  millimeters. 

Figure  9b  shows  the  reconstructed  image  using  the  linear  measurements  made  on  the  image  plane.  There 
is  a  clear  resemblance  between  the  reconstructed  image  and  the  raster  scan  image  of  Figure  9a.  The  technique 
used  to  reconstruct  the  image  is  described  in  detail  in.3,8  To  reconstruct  the  image  it  was  necessary  to  use 
regularization  when  inverting  the  measurement  matrix.  This  is  necessary  because  of  the  error  in  the  knowledge 
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Figure  9:  Images  of  the  object  in  Figure  8.  9a  Image  reconstructed  from  the  raster  scan,  9b  image  reconstructed 
from  the  linear  measurement. 


of  the  exact  positions  of  the  holes  on  the  disk  for  each  data  sample.  Analysis  shows8  that  the  quality  of  the 
reconstruction  will  improve  if  all  of  the  energy  that  goes  through  the  holes  is  measured.  Currently,  only  part  of 
the  energy  is  measured.  The  proper  way  of  collecting  all  of  the  energy  will  be  investigated  in  the  future. 

6.  FUTURE  EFFORTS 

Currently,  the  imager  can  perform  line  scans  at  rates  of  three  frames  per  second  and  the  data  is  post-processed 
to  reconstruct  the  images.  The  optical  system  was  designed  for  a  conjugate  point  scanning  imager  and  is  not 
appropriate  for  the  current  imaging  system.  In  the  current  implementation,  only  part  of  the  energy  that  goes 
through  the  holes  is  collected  by  the  receiver  antenna.  In  addition,  the  image  plane  mask  device  is  not  properly 
guarded  against  stray  radiation  from  the  different  reflective  materials  in  the  room.  We  plan  to  address  all  of 
these  issues  in  the  near  future. 

New  electronics  will  be  built  to  drive  the  hard  drive  spindle  motor  so  that  the  disk  can  be  spun  at  least 
30  revolutions  per  second.  This  will  facilitate  video  rate  imaging.  The  use  of  additional  down-converters  and 
an  analog  to  digital  conversion  card  interfaced  to  a  computer  will  facilitate  the  collection  of  the  measurement 
data  and  the  real  time  image  reconstruction.  We  are  in  the  process  of  designing  new  optics  for  the  imager  and 
illumination  systems.  This  will  facilitate  uniform  plane  wave  illumination  of  the  scene  and  the  minimization  of 
aberrations.  To  collect  all  the  energy  that  passes  through  the  holes  we  plan  to  build  a  lens  that  will  be  placed 
behind  the  holes.  The  lens  will  focus  the  energy  on  the  horn  of  the  receiving  antenna. 

One  of  the  factors  that  affect  the  quality  of  the  reconstructed  image  is  the  uncertainty  of  the  location  of 
the  holes.  To  mitigate  this,  we  will  use  precision  machining  to  drill  the  holes  at  known  locations.  This  and  the 
addition  of  the  lens  behind  the  holes  are  expected  to  improve  the  quality  of  the  reconstruction.  The  sensitivity 
of  the  imager  can  be  improved  if  the  image  plane  mask  is  shielded  properly  from  stray  radiation  that  makes  its 
way  into  the  receiver.  This  will  allow  us  to  form  images  of  objects  that  have  low  reflectivity  coefficients.  Also, 
by  increasing  the  size  of  the  imaging  window  we  can  form  larger  images.  Because  more  randomly  placed  holes 
will  be  contributing  to  the  measurement,  the  linearity  of  the  measurement  will  improve.  This  will  improve  the 
quality  of  the  reconstruction. 

The  imager  can  be  extended  to  full  two-dimensional  imaging  simply  by  replacing  the  slit  imaging  window  with 
a  rectangular  one  and  by  adding  more  randomly  placed  holes  on  the  disk.  The  image  reconstruction  technique  is 
the  same  as  for  the  line  image  and  has  been  described  in  detail  in  previous  publications.3,8  Preliminary  analysis8 
shows  that  we  will  be  able  to  generate  enough  random  patterns  in  one  revolution  of  the  disk  to  reconstruct  a  32 
by  32  pixel  image  from  linear  measurements.  When  the  line  imager  has  been  optimized  and  all  associated  issues 
have  been  resolved,  we  will  implement  the  full  two-dimensional  imager. 


7.  CONCLUSIONS 


We  have  presented  the  results  of  a  rigorous  three-dimensional  electromagnetic  analysis  and  used  them  to  design 
the  image  plane  mask.  For  the  available  disk  thickness  the  optimal  hole  radius  was  greater  than  1.4A.  We  also 
investigated  the  linearity  of  the  mask  and  concluded  that  the  mask  is  made  nearly  linear  if  the  scanning  holes 
are  placed  randomly  on  the  disk  and  separated  by  at  least  1A  from  each  other.  We  described  the  implementation 
of  the  system  in  detail.  Two  images  of  the  same  object  formed  using  the  line  imager  were  presented.  The  first 
image  was  formed  using  raster  scanning  as  the  object  was  moved  horizontally  on  a  translation  stage.  This  image 
was  used  as  a  reference.  The  second  image  was  reconstructed  from  linear  measurements  made  on  the  image  plane 
and  was  compared  against  the  first  image.  There  was  clear  resemblance  between  the  two  images.  Finally,  we 
discussed  our  approach  to  systematizing  the  device  so  that  the  images  can  be  scanned  and  reconstructed  at  video 
rates.  We  also  discussed  the  necessary  changes  that  need  to  be  made  to  improve  the  image  reconstruction  from 
linear  measurements. These  improvements  will  facilitate  the  successful  implementation  of  the  full  two-dimensional 
imager. 
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ABSTRACT 

In  this  paper  we  present  a  single  mode  active  device  for  sub- millimeter  wave  line  imaging.  The  illuminated  scene 
is  imaged  through  focusing  optics  onto  a  device  we  have  developed  and  have  dubbed  a  spatially  selective  mask 
(SSM).  This  device  transmits  parts  of  the  image  onto  a  heterodyne  receiver.  Currently  the  SSM  is  capable  of 
transmitting  user-selectable  parts  of  one  line  of  the  image  that  is  focused  on  it.  Multiple  patterns  are  used  to 
sample  a  line  in  the  image.  The  voltage  in  the  receiver  resulting  from  each  pattern  constitutes  an  independent 
measurement  of  the  illuminated  scene  along  a  line.  A  one  dimensional  image  is  reconstructed  from  the  mea¬ 
surement  results  and  a  priori  knowledge  of  the  patterns  using  methods  derived  from  the  theory  of  compressive 
sensing.  The  theory  behind  the  device  and  the  design  principles  we  use  are  reviewed.  We  show  line  images 
obtained  at  640  GHz.  Extension  of  this  technique  to  two  dimensional  imaging  is  discussed. 

1.  INTRODUCTION 

The  usefulness  of  sub-millimeter  wave  imaging  in  military  and  security  applications  needs  no  introduction.  Sub¬ 
millimeter  waves  can  penetrate  through  many  manmade  materials  and  provide  high  resolution  images  at  stand-off 
distances  while  still  not  suffering  tremendously  from  atmospheric  attenuation.1,2  However,  the  current  state  of 
the  art  is  limited  to  target  space  scanning  systems  that  require  complicated  mechanical  mechanisms  and  hence 
tend  to  be  bulky,  heavy,  and  not  portable.  These  systems  usually  make  use  of  a  single  pixel  detector  that  offers 
very  high  signal  to  noise  levels  due  to  the  maturity  of  the  technology.  Other  imaging  methods  that  perform 
well  in  other  modalities  such  as  interferometric  arrays  in  the  millimeter  wave  region3,4  or  focal  plane  arrays  in 
infrared  and  visible  regimes,  have  yet  to  be  demonstrated  in  the  sub- millimeter  wave  regime. 

Scanning  the  image  space  is  an  alternative  to  scanning  the  target  space.  The  image  space  is  smaller  than  the 
target  space  and  the  scanning  mechanisms  can  be  made  small,  thereby  eliminating  the  bulkiness  of  the  imager. 
Current  efforts  are  going  on  at  Rice  University  to  develop  a  electronically  controlled  mask  that  can  be  used  to 
scan  the  image  field.5  Although  promising,  they  have  not  been  able  to  achieve  the  high  modulation  levels  needed 
to  bring  the  signal  to  noise  ratio  to  acceptable  levels  for  image  formation.  We  have  developed  a  method  of  image 
space  scanning  with  high  modulation  levels,  high  scanning  rates,  low  mechanical  complexity,  and  low  cost.  At 
the  core  of  this  method  is  a  rotating  disk  with  holes  that  spins  at  a  constant  speed.  The  image  is  formed  on  one 
portion  of  the  disk  and  it  is  selectively  transmitted  through  the  holes  onto  the  receiver.  Linear  measurements 
are  made  on  the  image  as  the  disk  rotates.  An  image  is  reconstructed  on  a  screen  using  the  knowledge  of  the 
hole  patterns  and  the  measured  signal  from  the  receiver. 

The  paper  is  organized  as  follows.  Section  2  describes  the  device  and  the  method  of  image  reconstruction  in 
more  detail.  In  Section  3,  we  present  the  theoretical  formulation  and  numerical  methods  used  for  the  performance 
and  electromagnetic  analysis  of  the  device.  Results  of  the  analysis  that  pertain  to  the  design  of  the  device  are 
shown  in  Section  4.  The  image  simulation  results  and  the  experimental  image  results  are  shown  in  Section  5. 
Section  6  discusses  the  extension  of  the  imaging  methodology  to  two  dimensional  images.  Section  7  concludes 
the  paper. 
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Figure  1.  (Left)  Conceptual  sketch  of  the  line  imaging  device,  (right)  pixel  block  divisions. 


Imaging 


Motor 


2.  THE  SPATIALLY  SELECTIVE  MASK  AND  IMAGE  RECONSTRUCTION 

Figure  1  shows  a  conceptual  sketch  of  the  line  imaging  device  that  utilizes  a  spatially  selective  mask.  This  device 
is  placed  in  the  image  plane  of  the  imager  and  the  image  of  the  scene  is  formed  on  it.  The  device  consists  of  a 
rotating  disk  that  contains  holes  along  a  constant  radius.  The  image  is  formed  only  on  a  part  of  the  disk  and 
that  part  is  referred  to  as  the  imaging  window.  For  the  line  imager  this  window  resembles  a  slit.  The  receiver  is 
placed  behind  the  disk  and  all  the  energy  from  the  image  that  passes  through  the  holes  is  collected.  At  any  given 
time  several  holes  are  found  in  the  imaging  window.  These  holes  sample  the  image  that  is  formed  on  the  imaging 
window  by  allowing  only  the  energy  from  the  part  of  the  image  where  the  holes  are  located  to  be  measured.  A 
number  of  measurements  are  made  as  the  disk  is  rotated  and  the  pattern  of  holes  inside  the  imaging  window 
changes. 

In  order  to  reconstruct  the  image  from  these  measurements  the  imaging  window  is  divided  into  conceptual 
(imaginary)  blocks.  These  blocks  are  the  image  pixels  and  the  reconstruction  process  will  produce  the  average 
value  of  the  intensity  in  each  of  them.  If  the  number  of  measurements  made  is  less  then  the  number  of  pixels 
inside  the  imaging  window,  the  image  can  be  reconstructed  using  compressive  sensing  algorithms  as  presented 
in.6  Although  a  lot  of  research  is  being  dissipated  into  compressive  sensing  algorithms,  their  performance  with 
respect  to  time  is  still  below  that  required  for  video  rates  because  they  use  iterative  methods.  As  will  be  shown, 
one  of  the  advantages  of  our  technique  is  that  it  allows  us  to  take  as  many  linearly  independent  measurements  as 
there  are  pixels  in  the  image,  given  that  the  positions  of  the  holes  on  the  disk  have  been  carefully  chosen.  This 
advantage  can  be  exploited  in  the  reconstruction  process  as  presented  below. 

Our  imaging  problem  can  be  cast  into  a  matrix  equation  of  the  form 

rh  —  Mi  (1) 

where  m  is  a  vector  the  elements  which  contain  the  measurement  values  corresponding  to  each  pattern  of  holes 
in  the  imaging  window  as  the  disk  is  rotated.  M  is  a  full  rank  square  matrix  called  the  measurement  matrix. 
The  vector  i  contains  the  pixel  values  of  the  image  formed  on  the  imaging  window  and  is  the  unknown  quantity. 
Each  row  on  the  measurement  matrix  corresponds  to  one  pattern  of  holes  on  the  imaging  window  and  is  to  be 
registered  with  the  measurement  values  mn  corresponding  to  that  pattern.  Each  entry  in  the  rows  corresponds 
to  a  pixel.  The  numerical  value  of  each  entry  is  the  portion  of  the  pixel  corresponding  to  that  entry  that  is 
transmitted  through  the  hole. 

Assuming  that  the  holes  on  the  disk  are  placed  so  that  each  measurement  matrix  row  is  linearly  independent 
of  the  others  equation  1  can  be  solved  for  i  , 


i  —  M  m 


(2) 


Equation  2  can  be  solved  in  parallel  since  each  entry  of  i  can  be  computed  simply  through  addition  and 

multiplication  operations  on  the  elements  of  M  and  m.  This  is  very  important  when  the  technique  is  extended 
to  two  dimensions  where  the  number  of  computations  increases  greatly.  The  series  of  measurements  mn,  elements 
of  m,  can  be  collected  at  the  same  rotational  positions  of  the  disk  for  each  image  frame.  Hence,  the  elements  of 

Jd  need  to  be  computed  only  once.  Once  a  full  set  of  measurements  is  made  and  the  entries  of  the  measurement 
vector  m  are  known,  the  image  can  be  reconstructed  almost  instantaneously  by  solving  equation  2.  A  full  set  of 
measurements  can  be  collected  during  one  full  rotation  of  the  disk.  If  the  disk  is  rotated  at  a  constant  rate  of 
1800  rotations  per  minute  the  imager  will  produce  images  at  a  video  rates  of  30  lines  (or  frames)  per  second. 

For  the  line  imager  an  appropriate  pixel  block  can  have  the  shape  of  an  area  element  on  the  r  —  6  plane  of  a 
cylindrical  coordinate  system  as  shown  in  Figure  1  (right).  In  that  implementation  the  entries  of  the  measurement 
matrix  can  be  calculated  using  geometrical  formulas  given  that  we  know  the  centers  of  each  hole  (rm,  8m)  with 
respect  to  the  revolution  axis,  their  diameters,  and  the  angular  extent  of  the  pixels  ( 8P ).  Each  pattern  of  holes 
in  the  imaging  window  corresponds  to  one  row  of  the  measurement  matrix.  We  obtain  different  patterns  at 
different  rotational  positions  of  the  disk.  For  each  pixel  the  geometrical  formulas  are  used  to  determine  the  area 
of  intercept  of  that  pixel  with  the  holes  inside  the  imaging  window.  The  ratio  of  the  area  of  intercept  to  the  area 
of  the  pixel  becomes  the  numerical  entry  on  the  row  of  the  measurement  matrix  corresponding  to  that  pattern 
and  pixel. 


3.  ANALYSIS  METHODS 

It  is  natural  to  ask  several  questions  when  constructing  a  device  such  as  the  one  described  above.  What  is  the 
optimal  number  of  holes  inside  the  imaging  window  at  any  given  time?  What  is  the  minimum  dimension  of 
the  holes?  What  is  the  minimal  distance  between  holes?  What  is  the  optimal  depth  of  the  holes?  All  but  the 
first  question  require  a  rigorous  three-dimensional  electromagnetic  analysis.  The  first  question  is  answered  by 
observing  the  condition  number  of  the  measurement  matrix  as  we  change  the  number  of  random  holes  inside 
the  imaging  window.  The  condition  number  of  the  measurement  matrix  indicates  how  noise  is  amplified  in  the 
reconstruction  process  and  is  given  in  equation  3. 
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The  condition  number  is  always  greater  than  one,  with  one  being  the  lower  limit  signifying  a  well-conditioned 
matrix  and  infinity  signifying  a  singular  matrix.  Obviously  the  number  of  holes  that  can  be  placed  in  a  limited 
space,  i.e.  the  image  window,  is  determined  by  the  dimensions  of  the  holes  and  the  minimal  proximity  between 
any  two  holes. 


Transmission  through  the  cylindrical  holes  of  the  disk  has  characteristics  similar  to  transmission  through  a 
cylindrical  waveguide.7  The  transmission  depends  on  the  dimensions  of  the  cylindrical  hole.  The  waveguide 
behavior  of  the  cylindrical  holes  becomes  evident  when  the  thickness  of  the  disk  is  made  finite.  For  example, 
as  the  diameter  of  the  hole  is  increased  through  the  cutoff  value,  large  jumps  in  transmission  are  noticed  and 
then  the  transmission  decreases  until  the  cutoff  value  of  another  mode  is  reached  and  then  another  jump,  and 
so  on.  Also,  if  the  hole  diameter  is  kept  constant  and  the  thickness  of  the  disk  is  varied,  one  notices  sinusoidal 
variations  in  the  transmission  through  the  hole.  The  transmission  of  the  hole  for  hole  diameters  less  than  the 
cutoff  diameter,  decreases  exponentially  as  a  function  of  thickness.7  The  same  relationships  between  transmission 
through  the  aperture  and  aperture  dimensions  are  preset  with  thick  slits.8  Thick  slits  behave  like  parallel  plate 
waveguides  and  can  be  analyzed  using  a  two-dimensional  electromagnetic  analysis. 

Although  the  two-dimensional  electromagnetic  analysis  of  the  structure  would  reveal  the  same  relationships 
between  the  structure  parameters  and  the  power  transmitted  to  the  receiver,  it  can  not  be  used  to  obtain  design 
parameters.  The  reason  is  the  difference  between  the  cutoff  frequencies  of  a  parallel  plate  waveguide  and  the 
cutoff  frequencies  of  a  cylindrical  waveguide  (due  to  the  boundary  conditions).  The  design  parameters  can  be 
obtained  through  a  three-dimensional  analysis.  A  three-dimensional  analysis  is  being  implemented  and  will  be 
the  subject  of  another  publication. 


The  two-dimensional  analysis  can  give  us  insight  into  the  qualitative  behavior  of  our  imager  as  observed  by 
experiment.  For  this  purpose  we  implemented  a  two-dimensional  electromagnetic  simulation  using  a  Method  of 
Moments  technique.  The  geometry  for  the  simulations  is  shown  in  Figure  2.  The  width  of  the  slits  is  noted  by 
the  slit  separation  is  noted  by  d3 ,  the  slit  depth  is  noted  by  dj,  and  the  angle  of  incidence  of  the  incident 
field  is  noted  by  <p{  and  is  measured  with  respect  to  the  x-axis.  The  z-axis  is  pointing  out  of  the  page.  As  will 
be  delineated  next,  the  currents  on  the  contours  of  the  structure  are  the  primary  quantities  that  need  to  be 
calculated.  All  the  other  unknown  quantities  are  obtained  from  the  currents.  Since  the  contours  are  taken  on 
the  surface  of  the  structure  the  calculated  currents  are  also  physical  currents.  The  theoretical  formulation  of  the 
electromagnetic  equations  used  to  analyze  the  structures  of  interest  has  been  presented  previously.9  The  main 
equations  are  summarized  below. 

The  total  electric  field  at  any  point  in  space  is  given  by  the  sum  of  the  incident  and  scattered  fields 


*  (s)  =  (x)  +  (x)  (4) 

where  x  is  a  vector  in  the  x,  y  plane.  These  fields  satisfy  the  Helmholtz  wave  equations.  The  Incident  electric 
field  is  assumed  to  be  polarized  in  the  z-direction  and  is  propagating  towards  the  structure  in  the  negative 
y-direction.  The  scattered  field  satisfies  the  radiation  condition  in  all  directions,  while  the  incident  field  satisfies 
it  for  the  lower  half  space  since  it  is  incident  from  the  top  half  space.  For  several  reasons  made  clear  below,  the 
incident  field  is  modeled  as  a  superposition  of  tapered  Gaussian  beams  and  is  defined  as 

•  $i(i,i/)  =  y'an  ["  Wn(a- ^e“iko((*-*on)cos«Hv-vo„)  sin*)  da  (5) 

n  Jo 

where  k0  is  the  free  space  wave  number  for  the  wavelength  of  interest,  Onis  the  amplitude  of  the  nth  beam, 
xonand  yon  are  the  spatial  shifts  of  the  beam  on  the  structure,  and  Wn  (a)  is  the  Gaussian  spectrum  function  of 
the  nth  beam  and  is  given  by10,11 


W  (a)  = 


n  = 


2 

/cqt  sin  4>i 


The  tapering  factor  of  the  beam  is  represented  by  r  and  the  angle  of  incidence,  measured  with  respect  to  the 
positive  x  axis,  is  represented  by  (pi ,  as  shown  in  Figure  2.  Gaussian  tapering  is  convenient  for  several  reasons. 
It  makes  possible  the  calculation  of  the  transmitted  field  because  the  incident  field  is  almost  finite  in  extent  at 
the  structure.  It  minimizes  the  erroneous  effects  of  a  finite  structure  since  the  incident  field  at  the  ends  of  the 
structure  tends  towards  zero.  It  can  be  used  to  simulate  the  diffraction-limited  image  of  one  or  more  point  sources 
because  of  its  similarity  with  the  Sombrero  function.  This  allows  the  simulation  of  the  structure  of  interest  as 
part  of  an  imaging  system  and  is  very  useful  when  analyzing  the  resolution  properties  of  the  system. 


Using  Green’s  theorem  on  the  fields  and  the  Green’s  function  for  two  dimensions,  and  applying  the  boundary 
conditions  on  the  structure,  we  express  the  scattered  and  incident  fields  in  terms  of  the  currents  on  the  structure. 

The  far  field  for  the  incident  field  is  calculated  using  the  stationary  phase  approximation.12 
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The  scattered  field  in  the  far  region  is  given  by 
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where  Fs  (</>)is  the  scattering  amplitude,  5  =  xco s<f>s  +ysin<f>s  is  a  unit  vector  in  the  observation  direction 
and  <7(|x|)  is  the  propagation  phase  and  attenuation. 

Because  of  the  conservation  of  energy  the  power  of  the  incident  field  should  be  equal  to  the  sum  of  the 
reflected  power  and  transmitted  power. 

The  incident  power  is  given  by 


the  reflected  power  is  given  by 


Pi  =  -  f  \FM)\2d<t> 
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and  the  transmitted  power  is  given  by 

Pt=-  I**  \Fi(4>)  +  Fs(4>)\2d4> 

VO  J* 


(12) 
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The  transmission  power  coefficient  is  calculated  as  the  ratio  of  the  transmitted  power  to  the  incident  power. 

A  numerical  solution  for  the  currents  can  be  obtained  using  the  Method  of  Moments.  The  details  of  the  MoM 
solution  have  been  presented  previously  in.9  Once  the  currents  are  known  we  can  calculate  the  far  scattered 
field  using  the  discreet  form  of  equation  10.  The  far  incident  field  is  calculated  directly  from  equation  7.  The 
powers,  hence  the  transmission  coefficient,  can  be  calculated  using  equations  12,  13,  and  14,  once  the  far  fields 
are  known. 
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Figure  3.  Ratio  of  well-conditioned  measurement  matrices  to  total  number  of  measurement  matrices 


4.  SIMULATION  RESULTS  AND  DISCUSSION 

To  investigate  the  performance  of  the  reconstruction  process  as  a  function  of  the  number  of  holes  inside  the 
imaging  window,  we  simulated  the  scenario  illustrated  in  Figure  l(right).  The  image  window  is  divided  into 
twenty  pixels  of  equal  angular  extent  <f>p  =  n/200 ,  the  angular  extent  of  each  hole  was  about  7r/170,  corresponding 
to  a  hole  diameter  of  1  mm  positioned  at  a  radial  distance  from  the  center  of  revolution  of  about  58  mm.  The 
holes  were  positioned  at  a  random  angular  distance  from  each  other  on  a  constant  radius.  The  random  angular 
distance  was  generated  as  follows 


0S  =  n$m  +  ©  (15) 

where  0a  is  the  angular  separation  distance,  0m  is  the  angular  extent  of  the  holes,  n  is  a  number  between  0.1 
and  6,  ©  is  a  uniformly  distributed  random  variable  that  takes  values  between  O.10m  and  6dm •  The  term  ndm 
determines  the  minimum  angular  separation  distance  between  two  holes.  A  sequence  of  30  holes  was  generated. 
Several  measurement  matrices  were  calculated  as  the  patterns  moved  across  the  imaging  window  and  their 
condition  number  was  recorded.  Each  measurement  matrix  had  dimensions  of  20  by  20.  For  a  given  minimum 
separation  distance  not  all  of  the  calculated  measurement  matrices  were  well-conditioned.  We  calculated  the 
ratio  of  well-conditioned  measurement  matrices  to  the  total  number  of  measurement  matrices  for  each  minimum 
separation  distance  n#m.  The  results  are  shown  in  Figure  3. 

We  defined  a  well-conditioned  matrix  to  have  a  condition  number of  less  than  500.  Although  this  is  a  large 
condition  number  it  is  less  than  half  the  average  condition  number  of  a  20  by  20  matrix  with  random  elements. 
In  addition  the  mean  «  of  the  matrices  with  k  smaller  than  500  was  around  90  for  all  of  the  minimum  separations, 
more  than  10  times  better  than  a  matrix  of  random  coefficients. 

The  plot  in  Figure  3  shows  that  the  number  of  well-conditioned  measurement  matrices  that  can  be  generated 
given  a  random  placement  of  holes  is  larger  when  the  separation  between  adjacent  holes  is  small,  i.e.  the  number 
of  holes  in  the  imaging  window  is  large.  This  result  is  to  be  expected  since  the  variation  in  the  patterns  increases 
as  the  total  perimeter  length  of  the  the  holes  inside  the  imaging  window. 

For  the  line  imager  we  only  need  to  have  one  measurement  matrix  so  the  result  above  might  seem  unnecessary 
given  that  a  well-conditioned  measurement  matrix  can  be  found  for  any  value  of  n#m>  however  for  a  two  dimen¬ 
sional  imager  the  result  of  Figure  3  is  important.  It  demonstrates  the  ability  of  the  rotating  disk  to  implement 
equation  1,  since  many  linearly  independent  measurement  matrix  rows  can  be  generated  in  one  rotation. 

One  infers  from  the  result  of  Figure  3  that  the  number  of  possible  well-conditioned  measurement  matrices 
can  be  increased  by  decreasing  the  size  of  the  holes  so  that  more  of  them  can  be  placed  in  the  imaging  window. 
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Figure  4.  Structures  for  the  simulation  of  the  non-linearity  of  the  mask 


However,  both  the  size  of  the  holes  and  their  minimum  separation  distance  are  limited  by  the  physics  of  the 
structure.  As  it  is  shown  in7  and  in13  choosing  a  smaller  hole  diameter  limits  transmission  through  the  hole  and 
some  diameters  have  better  transmission  than  others.  Also,  as  the  holes  are  placed  closer  to  each  other,  mutual 
coupling  between  the  holes  adversely  affects  the  linearity  of  the  mask  assumed  by  equations  1  and  2.  These  same 
effects  are  observed  for  thick  slits.8 

The  two  dimensional  analysis  presented  in  Section  3  pertains  to  a  slit  geometry.  Therefore,  it  cannot  be  used 
to  obtain  design  parameters  for  our  device,  such  as  the  diameter  of  the  holes,  the  thickness  of  the  disk,  or  the 
minimal  separation  distance  for  which  the  linearity  is  still  preserved.  We  can  however,  at  least  qualitatively, 
observe  the  effect  of  the  hole  separation  distance  on  the  non-linearity  of  the  measurement. 

We  simulated  the  linearity  of  a  slit  mask  for  the  configuration  shown  in  Figure  4.c.  The  transmitted  power 
from  the  structures  in  Figures  4.a.  and  4.b.  was  calculated  using  equation  14.  The  sum  of  the  two  was  compared 
with  the  power  transmitted  from  the  structure  in  Figure  4.c.  The  depth  of  the  slits  d was  set  to  2.7A,  the  width 
of  the  slits  dw  was  set  to  2.1  A,  and  the  separation  distance  was  varied  from  0.1  A  to  10A.  These  values  were 
chosen  to  match  the  dimensions  of  the  experimental  setup.  We  could  have  selected  some  other  values  since  the 
two  dimensional  simulation  is  only  qualitatively  and  not  quantitatively  equivalent  to  the  physical  structure  with 
holes.  The  extent  of  the  structure  was  40 A.  The  incident  field  was  normally  incident  (<f>i  =  7t/2)  with  a  tapering 
factor  r  =  5A  and  spatially  centered  at  the  origin. 

non-linearity  as  a  function  of  separation  distance  non-linearity  as  a  function  of  separation  distance 


Figure  5.  Results  of  the  linearity  analysis  (left)  when  a 11  of  the  transmitted  power  is  compared;  (right)  when  the  transmitted 
power  between  262.8  and  277.2  degrees  is  compared. 


Figure  5  shows  the  results  of  the  linearity  analysis.  On  the  right  side  we  have  plotted  the  percent  error 
when  the  total  power  transmitted  through  the  structure  is  collected.  This  is  the  linearity  error  and  is  computed 
by  taking  the  difference  of  the  transmitted  powers  and  normalizing  them  by  the  sum  of  the  powers  from  the 
structures  a  and  b.  The  difference  is  very  small  with  a  maximum  of  two  tenths  of  a  percent.  We  notice  that  the 


error  oscillates  around  zero  with  some  separations  being  linear  (error  is  approximately  zero)  and  some  not.  The 
period  of  oscillations  is  one  wavelength.  The  cusps  are  numerical  artifacts  due  to  the  large  condition  number  of 
the  MoM  matrix  for  the  given  geometry  of  structure  c.  The  linearity  error  decreases  as  the  separation  of  the  slits 
increases  indicating  that  the  slits  decouple.  In13  the  authors  conclude  that  the  coupling  between  holes  becomes 
insignificant  when  the  ratio  of  the  separation  , measured  from  the  centers  of  the  holes,  to  the  radius  of  the  holes 
is  about  six. 

Figure  5(right)  shows  the  linearity  error  when  only  the  power  between  262.8  and  277.2  degrees  of  the  far 
field  is  collected.  This  error  is  extremely  large  but  decreases  as  the  distance  between  separations  increases.  This 
result  is  expected  since  the  sum  of  the  powers  transmitted  through  structures  a  and  b  ignores  the  interference 
pattern  of  the  two  slits  and  the  far  field  pattern  will  be  a  Gaussian.  The  power  transmitted  through  structure  c 
will  display  an  interference  pattern  in  the  far  field.  Therefore,  the  transmitted  powers  will  differ  in  a  particular 
section.  As  the  separation  distance  increases  the  zeros  of  the  interference  pattern  come  closer  together  and  the 
power  is  concentrated  more  and  more  in  the  center  of  the  far  field,  and  the  difference  between  the  transmitted 
powers  decreases. 

Using  the  linearity  results  presented  in  Figure  5(left)  we  can  construct  a  image  scanning  scenario  where 
the  majority  of  the  holes  are  spaced  such  that  the  mask  is  non-linear  (worst  case  scenario).  In  the  following 
discussions  we  will  refer  to  the  slits  in  the  simulation  as  holes.  In  order  to  have  a  reference  to  compare  the 
reconstructions  with,  a  singe  hole  was  placed  at  some  distance  from  the  others  (approximately  40A)  so  that  we 
could  simulate  raster  scanning.  The  structure  length  was  40A.The  field  incident  on  the  structure  was  composed 
of  two  normally  incident  beams  with  tapering  factor  of  r  =  3  and  spatial  centers  at  x\  =  — 5A,  =  0  and 

x2  =  3A,  3/2  =  0. 
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Figure  6.  Results  of  the  image  reconstruction  simulation  using  power  measurements;  (left)  when  all  of  the  transmitted 
power  is  collected;  (right)  when  the  transmitted  power  between  262.8  and  277.2  degrees  is  collected. 


The  results  of  this  simulation,  when  the  measurement  data  used  in  the  reconstruction  consists  of  all  of  the 
transmitted  power,  are  shown  in  Figure  6(left).  The  top  image  in  Figure  6(left)  shows  the  reconstructions  for 
different  measurement  matrices  using  the  corresponding  measurement  data.  Each  column  of  the  image  represents 
a  reconstruction  of  the  line  image.  To  the  left  we  see  the  result  of  the  raster  scan.  The  image  is  normalized 
by  the  maximum  of  the  raster  scan  image.  The  white  space  that  separates  the  raster  scan  images  from  the 


reconstructed  images  corresponds  to  the  part  of  the  scan  that  did  not  contain  holes,  hence,  the  measurement 
matrix  was  singular.  The  bottom  plot  shows  the  condition  number  of  the  measurement  matrix,  the  negative 
values  correspond  to  condition  numbers  greater  than  104.  The  reconstruction  is  generally  very  good  and  the  worst 
reconstruction  cases  correspond  to  measurement  matrices  with  a  very  large  condition  number.  The  reconstruction 
however  is  not  perfect  and  this  is  attributed  to  the  non-linearity  of  the  measurements. 

If  the  measurement  data  used  for  the  reconstruction  consists  only  of  part  of  the  transmitted  power,  the 
reconstruction  deteriorates.  Figure  6 (right)  shows  the  results  of  this  reconstruction.  The  measurement  data 
consists  of  the  transmitted  power  collected  between  262.8  and  277.2  degrees  of  the  far  field.  Some  reconstructions 
are  qualitatively  similar  to  the  raster  scan.  The  reason  for  those  reconstructions,  although  the  linearity  error  as 
shown  in  Figure  5(right)  is  large,  is  attributed  to  the  presence  of  more  than  two  holes  diffracting  the  incident 
energy.  The  main  lobe  from  the  interference  pattern  of  more  than  two  holes  is  more  narrow  than  the  one  for  two 
holes.  Also,  more  of  the  energy  is  concentrated  in  the  main  lobe.  Hence  the  collected  energy  represents  more  of 
the  main  lobe  and  the  main  lobe  represents  more  of  the  total  energy.  Therefore,  the  non-linearity  error  is  less 
than  the  error  in  Figure  5 (right).  As  it  is  to  be  expected  the  reconstructions  from  the  measurement  matrices 
with  larger  condition  number  are  affected  the  most. 

5.  EXPERIMENTAL  MEASUREMENTS  AND  RESULTS 

Using  scrounged  materials  we  built  a  line  imager  in  our  lab.  The  imager  and  its  components  are  shown  in  Figure 
7.  We  used  the  spindle  motor  and  disk  from  a  5.25  inch  Quantum  Bigfoot  hard  drive.  Holes  were  drilled  on 
the  disk  at  a  constant  radius  of  58  mm.  A  1  mm  drill  bit  was  used  to  drill  15  holes  spaced  at  random  with 
separation  distances  between  2.3  mm  and  5.3  mm.  Another  hole  was  drilled  about  50  mm  arc  away  from  the 
others.  This  single  hole  was  used  for  raster  scanning.  The  disk  was  1.27  mm  thick.  We  removed  the  top  cover 
and  cut  a  rectangular  aperture  on  the  back  of  the  hard  drive  case  so  that  the  signal  could  be  let  into  the  receiver. 
The  motor  was  rotated  at  a  constant  velocity  of  180  revolutions  per  minute  so  that  about  750  measurements  are 
captured  for  one  revolution.  A  photo-diode  detector  was  placed  in  the  proximity  of  the  edge  of  the  disk.  A  piece 
of  dark  tape  was  placed  on  the  edge  of  the  disk  to  obtain  a  pulse  for  every  revolution.  The  pulse  was  used  to 
trigger  a  sweep  of  the  spectrum  analyzer  (Agilent  SCA  Spectrum  Analyzer  AN1996A  )  so  that  we  could  register 
the  measurements  with  the  positions  of  the  holes.  The  operating  frequency  of  the  source  receiver  pair  was  640 
GHz.  The  signal  from  the  receiver  was  down-converted  to  4.8  GHz  and  supplied  to  the  spectrum  analyzer.  The 
optical  system  consisted  of  an  elliptical  mirror  with  one  focus  at  1  m  and  the  other  at  10  m.  The  line  imager 
was  placed  on  the  1  m  side  of  the  mirror  with  the  receiver  positioned  approximately  100  mm  behind  the  rotating 
disk.  The  target  consisting  of  a  mask  in  front  of  the  receiver  was  placed  on  the  side  of  the  10  m  focus.  The  source 
was  placed  approximately  0.5  m  behind  the  mask.  The  mask  consisted  of  two  holes  cut  in  a  piece  of  carpet.  The 
image  of  the  mask  was  focused  on  the  line  imager  and  a  sweep  of  data  was  saved  manually  onto  a  flash  drive 
from  the  spectrum  analyzer.  The  saved  data  was  post-processed. 


Figure  7.  Picture  of  the  fine  imager. 
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Figure  8.  (Left) results  of  the  image  reconstruction  using  experimental  data;  (right)  corresponding  measurement  data 

Figure  8(left)  shows  the  results  of  the  reconstruction.  The  measurement  data  used  for  the  reconstruction 
are  shown  to  the  right.  We  used  regularization  when  calculating  the  inverse  measurement  matrix.  This  was 
necessary  to  obtain  decent  reconstruction.  Just  like  with  the  simulation  results  shown  in  Figure  6,  each  column 
of  the  image  on  the  top  (left)  figure  represents  a  reconstruction  of  the  line  image.  The  partial  reconstructions 
that  give  the  appearance  of  a  slant  are  due  to  the  use  of  regularization. 

The  quality  of  the  reconstruction  can  be  attributed  to  a  couple  of  factors.  Only  part  of  the  power  is  collected 
therefore  the  reconstruction  is  noisy.  These  results  agree  with  the  simulations  presented  in  Section  4.  Another 
source  of  error  was  the  wobbling  of  the  disk,  although  small.  Yet  another  source  of  error  was  the  lack  of  accuracy 
in  the  knowledge  of  the  exact  positions  of  the  holes  on  the  disk,  leading  to  inaccurate  measurement  matrices. 
The  holes  on  the  disk  were  marked  at  the  desired  locations  and  drilled  on  a  12  inch  drill  press  while  the  disk  was 
held  on  the  bench  by  hand.  Needless  to  say  this  was  not  a  precise  manufacturing  process,  and  it  is  a  wonder 
that  we  can  reconstruct  the  image  at  all. 

It  was  observed  (results  not  shown  here)  that  the  quality  of  the  reconstruction  improved  when  the  diameter 
of  the  holes  was  increased  to  2  mm.  This  can  be  attributed  to  the  increased  signal  (more  power  was  let  through 
the  holes)  and  to  the  fact  that  the  larger  holes  diffract  less  than  the  small  ones,  so  the  power  was  directed  more 
towards  the  receiver. 


6.  EXTENSION  TO  TWO  DIMENSIONS 

Figure  9  shows  a  conceptual  sketch  of  the  spatially  selective  mask  device  for  imaging  in  two  dimensions.  Equa¬ 
tion  1  is  valid  for  both  the  one-dimensional  and  two-dimensional  imaging  problems.  In  the  line  image  setup 
the  arranging  of  the  measurement  matrix  row  elements  was  obvious.  For  the  two  dimensional  image  the  two 
dimensional  array  of  pixels  is  linearized;  each  row  of  the  array  is  concatenated  to  the  end  of  the  previous  row. 
Just  like  for  the  line  imager  a  full  set  of  measurements  can  be  collected  during  one  full  rotation  of  the  disk. 
Therefore,  if  the  disk  is  rotated  at  a  constant  rate  of  1800  rotations  per  minute  the  imager  will  produce  images 
at  a  video  rates  of  30  frames  per  second. 

An  appropriate  pixel  block  for  this  implementation  would  be  a  square.  In  that  case  the  entries  of  the 
measurement  matrix  can  be  calculated  by  the  following  method.  The  imaging  window  is  divided  into  conceptual 


Pignre  9.  (Left)  Conceptual  sketch  of  the  two  dimensional  imaging  device,  (right)  pixel  block  divisions 


square  blocks  as  shown  in  Figure  9(right),  each  corresponding  to  an  image  pixel.  Each  pattern  of  holes  in  the 
imaging  window  corresponds  to  one  row  of  the  measurement  matrix.  We  obtain  different  patterns  at  different 
rotational  positions  of  the  disk.  For  each  pixel  a  Monte  Carlo  integration  method  is  used  to  determine  the  area 
of  intercept  of  that  pixel  with  the  holes  inside  the  imaging  window.  The  ratio  of  the  area  of  intercept  to  the  area 
of  the  pixel  is  the  entry  on  the  row  of  the  measurement  matrix  corresponding  to  that  pattern  and  pixel. 

7.  CONCLUSIONS 

We  have  shown  simulation  and  experimental  results  supporting  a  method  of  image  space  scanning  with  high  mod¬ 
ulation  levels,  high  scanning  rates,  low  mechanical  complexity,  and  low  cost.  We  observed  that  randomly  placing 
the  holes  close  to  each  other  and  having  many  of  them  in  the  imaging  window,  produces  more  well-conditioned 
measurement  matrices  than  does  placing  them  far  and  having  less  of  them  in  the  imaging  window.  However,  as 
the  holes  come  closer  to  each  other  non-linearity  effects  are  observed.  Also  the  transmission  through  the  holes  is 
limited  by  the  waveguide- like  behavior.  Simulations  of  linear  measurements  on  a  image  were  performed  using  a 
two  dimensional  electromagnetic  analysis  approximating  the  holes  with  infinite  slits.  Although  the  electromag¬ 
netic  analysis  could  not  be  used  to  design  the  device  it  satisfactorily  explained  the  experimental  observations 
qualitatively.  From  the  results  of  Figures  6  we  conclude  that  for  the  best  performance  we  need  to  collect  all 
of  the  energy  that  is  transmitted  through  the  holes.  The  same  image  formation  methodology  used  for  the  line 
imager  can  be  extrapolated  to  a  two-dimensional  imager  and  we  have  introduced  the  required  adaptations  to  do 
so. 
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Abstract 

The  performance  of  a  binary  output  beam  break  sensor  is  characterized  by  its  information  rate  per  pixel, 
A  physics  based  model  utilizes  target  and  noise  spectra  to  establish  the  channel  capacity  and  the 
realizable  rate.  This  model  enables  the  determination  of  a  system  of  sensors  performance  metric, 
suitable  for  optimizing  system  performance,  A  method  of  optimizing  sensor  design  within  the  system  of 
sensors  is  discussed.  Numerical  results  from  the  model  illustrate  the  process  for  a  beam  break  sensor 
such  as  one  used  on  garage  door  openers.  Also,  the  numerical  results  coupled  with  classification 
performance  data  may  quantify  the  value  of  information  for  such  a  sensor,  as  discussed  in  the  paper. 

An  uncertainty  analysis  is  presented  for  a  beam  break  sensor  used  to  estimate  a  target's  height  to  width 
ratio,  useful  for  human/other  classification.  The  uncertainty  analysis  suggests  some  quantitative 
method  of  determining  the  minimum  number  of  pixels  desired. 

Introduction 

Beam  break  sensors  are  an  example  of  a  class  of  sensors  that  provide  profile  information  about  a  target 
[Russomanno,  2010].  Such  profiling  sensors  may  provide  sufficient  information  for  some  applications  or 
they  may  be  part  of  a  system  of  sensors.  It  is  important  for  optimizing  performance  of  a  system  of 
sensors  that  a  system  performance  metric  be  available.  Such  a  metric  enables  system  performance 
optimization  for  many  aspects  of  the  system.  Examples  include  sensor  selection  from  a  suite  of 
available  sensors,  sensor  design  within  given  types  of  sensors  for  the  network  or  algorithm  choices  for 
automatic  processing  of  some  sensor  outputs. 

In  general,  systems  of  sensors  are  likely  to  utilize  automated  processing  at  a  variety  of  levels.  Such 
processing  might  be  in  the  form  of  image  enhancement,  image  fusion  for  human  observers  or  classifiers 
that  provide  cues  for  human  observers.  This  paper  on  profiling  sensors  considers  the  use  of  a  classifier 
but  does  not  consider  image  fusion  or  enhancement. 

Two  metrics  are  often  used  for  imaging  sensors  [Driggers,  1998].  NIIRS  (National  Imagery  Information 
Rating  Scale)  is  often  used  in  surveillance  applications,  and  the  Targeting  Task  Performance  metric  is 
often  used  for  tactical  imagers.  An  information  theoretic  metric  (information  rate  in  bits/pixel)  is  also 
commonly  used.  At  their  foundations  lie  the  same  sensor  parameters,  resolution  and  signal  to  noise 
ratio.  All  are  also  active  areas  for  research  [Irvine,  2007,  Westover,  2008,  Vollmerhausen,  2006].  Each 


has  its  own  domain  of  applicability,  with  some  overlap.  For  the  beam  break  sensor  presented  here,  we 
use  information  rate  as  the  metric  for  modeling  the  beam  break  profiling  sensor.  One  reason  is  that 
information  rate  does  not  implicitly  incorporate  a  model  for  the  human  observer.  However,  some 
researchers  have  modeled  human  observer  performance  in  terms  of  information  rate.  This  flexibility 
allows  one  to  model  algorithms  and  human  observers  within  a  system.  Furthermore,  the  point  of  view 
is  not  from  above  (as  with  NIIRS)  and  the  task  is  not  quite  in  the  realm  of 

detection/recognition/identification  (as  with  Targeting  Task  Performance  metric).  So  among  existing 
metrics  information  rate  is  the  metric  immediately  applicable  to  profiling  sensors  as  part  of  a  system. 
Imaging  sensors  and  other  types  of  sensor  can  also  be  modeled  using  information  rate  as  a  metric  within 
the  information  theoretic  framework  [Huck,  1996].  This  facilitates  system  level  modeling  by  utilizing  a 
common  metric.  Also  the  end  numerical  results  in  bits  per  pixel  have  a  more  general  understanding 
than  most  other  metrics. 

In  summary  the  work  presented  in  this  paper  represents  a  significant  step  in  the  direction  of  modeling  a 
system  of  sensors  that  includes  sensors  of  various  types.  These  types  are  likely  to  include  an  imaging 
sensor  supplemented  perhaps  by  profiling,  radar,  seismic  and  acoustic  sensors  as  well  as  intelligence 
information.  We  expect  the  system  of  sensors  to  include  automated  processing  of  sensor  output  as  well 
as  human  observers  using  imagery.  Specifically  the  paper  presents  the  modeling  process  for  the 
information  rate  of  a  binary  output  beam  break  profiling  sensor.  Numerical  results  illustrate  the  model. 
Additional  discussions  show  how  the  results  are  important  to  sensor  system  modeling  and  optimization. 

Binary  Output  Beam  Break  Sensor 

As  with  imaging  sensors  there  are  many  configurations  of  profiling  sensors.  Each  has  its  own  unique 
characteristics  as  well  as  sharing  characteristics  with  a  general  profiling  sensor.  The  binary  output  beam 
break  sensor  is  chosen  as  being  representative  of  the  broader  class  of  profiling  sensors.  The  model 
presented  here  captures  the  more  general  aspects  of  profiling  sensors.  The  model  has  sufficient  fidelity, 
accuracy  and  complexity  to  illustrate  the  behavior  of  the  sensor  within  a  system  of  sensors.  This  desire 
for  a  system  level  model  leads  to  an  information  theoretic  model.  Any  sensor  can  be  modeled  from  that 
viewpoint,  making  information  rate  a  common  metric  for  any  sensor  in  the  network. 

A  binary  output  beam  break  sensor  has  a  two  state  output  indicating  that  a  beam  is  blocked  or  not 
blocked.  It  was  proposed  as  a  low  cost  sensor  that  could  distinguish  humans  from  animals  by  assessing 
their  profiles.  Several  other  versions  of  profiling  sensors  have  been  built,  generally  passive  versions. 

The  two  state  profiles  can  also  result  from  applying  a  threshold  to  detector  outputs  with  greater  bit 
depths.  However,  the  original  beam  break  sensor  was  inspired  by  the  common  garage  door  safety 
detector,  and  the  notional  sensor  modeled  here  is  that  type  of  profiling  sensor.  It  consists  of  a 
transmitter  module  and  a  detector  module  whose  output  is  either  high  or  low.  We  arbitrarily  choose 
the  presence  of  the  target  to  be  a  high  state  and  the  absence  of  the  target  to  be  the  low  state. 

In  its  simplest  implementation  the  transmitter/detector  module  pair  is  replicated  in  the  vertical 
direction,  creating  a  linear  array  of  detectors.  Other  deployments  are  also  possible.  The  binary  detector 
outputs  are  sampled  in  time.  The  target  must  move  through  the  beams  to  create  its  profile.  Targets  are 
assumed  to  move  in  the  horizontal  direction,  breaking  the  separate  beams  as  the  target  passes  through 


the  sensor.  The  temporal  sampling  coupled  with  the  spatial  sampling  in  the  vertical  direction  produces 
data  that  is  easily  interpreted  as  a  two  dimensional  target  profile. 

Each  detector  and  transmitter  has  its  own  field  of  view.  Figure  1  depicts  a  general  situation  where  the 
detector  and  transmitter  fields  of  view  differ.  For  our  example  sensor  the  two  are  the  same.  The  figure 
shows  a  slice  of  the  three  dimensional  field  of  view  cones.  That  means  the  vertical  arrows  are  slices  of 
an  edge.  For  discussion  we  choose  to  think  of  Figure  1  as  a  top  view.  That  means  a  vertical  edge  is 
being  moved  horizontally  into  the  field  of  view  (at  one  of  three  locations  in  Figure  1.)  For  an  analog 
system  the  detector  response  diminishes  as  soon  as  the  edge  reduces  the  received  radiation.  Figure  2 
shows  a  simulated  analog  edge  response  with  added  noise.  It  also  demonstrates  the  one-to-one 
relationship  between  edge  position  within  the  field  of  view  and  the  analog  detector  output.  A  binary 
output  sensor  does  not  show  any  change  until  the  moving  edge  has  blocked  a  threshold  amount  of 
radiation.  Fifty  percent  blockage  is  a  convenient  threshold  since  the  output  change  occurs  with  the  edge 
at  or  near  the  optical  axis.  Therefore,  the  binary  output  toggles  as  the  edge  reaches  the  optical  axis. 
Figure  1  illustrates  that  the  amount  of  edge  movement  within  the  analog  edge  response  varies  with  the 
edge  location  between  the  transmitter  and  the  detector.  It  also  shows  that  the  maximum  field  of  view 
has  an  upper  bound  of  the  larger  angular  field  of  view  times  half  the  separation  distance  between 
transmitter  and  receiver. 

The  geometrical  complications  are  minimized  in  our  example  system  which  has  equal  detector  and 
transmitter  fields  of  view.  And  we  choose  to  locate  the  target  edge  midway  between  the  transmitter 
and  receiver.  A  fifty  percent  threshold  is  also  assumed.  These  conditions  match  the  experimental 
conditions  for  the  garage  door  opener  safety  switch  used  to  obtain  numerical  results. 

In  an  array  of  such  transmitter/detector  pairs,  the  fields  of  view  of  adjacent  pairs  may  overlap.  This 
overlap  increases  the  transmitted  power  seen  by  any  particular  detector.  The  additional  power  from 
other  transmitters  varies  with  the  configuration  chosen  for  the  array.  Detectors  near  the  middle  of  a 
linear  vertical  array  (our  notional  sensor  array)  receive  extra  optical  radiation  from  transmitters  above 
and  below.  Those  detectors  near  the  top  or  bottom  receive  radiation  that  is  not  symmetrical  in  the 
vertical  direction.  Regardless  of  the  total  received  power,  the  binary  output  sensor  toggles  when  the 
power  is  reduced  by  fifty  percent  of  its  unblocked  value.  Therefore  our  notional  model  assumes  the 
detector/transmitter  pairs  are  isolated  from  the  other  pairs  that  form  the  profiling  sensor  array. 

The  sensor  output  can  be  used  as  input  to  an  automated  classification  algorithm  [Chari,  2008.] 

However,  viewing  it  in  an  image  format  provides  insight  to  typical  uses.  Figure  3  shows  three  profile 
images  for  illustration  purposes  and  to  help  understand  the  validity  of  assumptions  made  in  the 
analytical  modeling  process  that  follows. 

Model  for  Binary  Output  Beam  Break  Sensor 

The  discussion  above  suggests  the  following  characteristics  for  the  binary  output  sensor  and  targets  in 
our  model.  Sampling  in  the  vertical  direction  is  achieved  by  separate  transmitter/detector  pairs. 

Sampling  in  the  horizontal  direction  occurs  by  temporally  sampling  the  binary  output  state  of  the 
detectors.  Block  targets  move  only  horizontally  while  passing  the  vertical  line  of  sensors.  The  detectors 


change  states  when  the  block  obscures  half  of  the  optical  power  being  received  by  the  detectors  with  no 


obscuration.  Targets  too  small  to  block  half  of  the  received  radiation  will  not  be  detected  by  the 
profiling  sensor  and  result  in  no  information  from  the  sensor.  We  arbitrarily  choose  the  blocked  state  to 
be  binary  output  1  and  the  unblocked  state  to  be  0.  The  threshold  response  changes  the  analog  edge 
response  of  Figure  2  to  a  sharp  transition  from  0  to  1  at  the  optical  axis  (0.5  position  in  Figure  2). 

These  properties  imply  a  two  dimensional  output  from  the  array,  a  spatial  output  in  the  vertical 
direction  and  a  temporal  output.  Instead  of  using  spatial/temporal  pixels,  the  horizontal  motion  of  the 
target  coupled  with  the  temporal  sampling  is  modeled  as  a  spatial  pixel.  Pixel  size  in  the  horizontal 
direction  is  determined  by  the  sample  rate  and  the  speed  of  the  target.  In  effect  the  model  equivalently 
locates  a  stationary  target  within  a  horizontal  field  of  view.  The  equivalent  horizontal  field  of  view  is 
determined  by  the  speed  of  the  target,  the  sample  rate,  and  the  period  of  time  being  considered.  The 
period  is  conveniently  chosen  to  include  both  edges  of  the  target.  Most  often  this  sensor  is  used  for 
automated  human/other  classification.  Therefore,  numerical  examples  will  draw  from  this  scenario. 

Information  Rate  Equivalents  for  Sensors 

Information  rate  as  we  use  the  term  does  not  include  the  actual  communication  channel  used  to 
transmit  information  from  the  sensor  to  the  user.  Models  for  the  communication  channel  are  highly 
developed  and  therefore  not  addressed  here.  Our  interest  is  in  the  sensor's  effect  on  the  information  in 
a  spatial  scene  as  that  spatial  information  is  converted  from  optical  radiation  to  an  electronic  signal. 
Consequently,  spatial  frequency  domain  techniques  provide  the  tools  for  characterizing  the  information 
capabilities  of  imaging  sensors  and  profiling  sensors.  The  two  dimensional  character  of  the  information 
is  modeled  by  assuming  separable  dimensions. 

The  major  factors  in  information  transfer  by  an  imaging  sensor  include  blur,  noise  and  dynamic  range. 

In  addition  to  imaging  blur,  blur  is  sometimes  used  to  model  atmospheric  effects  such  as  turbulence. 

Blur  is  also  intimately  linked  to  resolution  which  leads  to  the  critical  parameter  of  pixels  on  the  target. 
Noise  arises  from  a  number  of  sources,  but  includes  detector  noise,  temporal  quantization  noise,  spatial 
sampling  noise  and  can  include  speckle  for  laser  illuminated  scenes.  Noise  at  the  system  level  is  affected 
by  temporal  factors  such  as  frame  rate/integration  time.  Dynamic  range  can  be  affected  by  blur,  but  is 
also  impacted  by  pedestal  removal  and  automatic  gain  control.  Many  modern  imaging  sensors  use 
sophisticated  algorithms  to  process  images  and  usually  include  automated  control  of  the  dynamic  range. 
These  factors  must  be  included  in  our  model  for  the  binary  output  beam  bread  sensor.  Some  are 
treated  by  our  assumptions  that  establish  the  domain  of  applicability  for  the  information  theoretic 
model.  For  example  we  assume  the  noise  to  be  additive  white  noise. 

Traditional  imaging  sensors  with  focal  plane  arrays  for  detectors  are  well  modeled  by  the  information 
rate  metric  [Huck,  1996].  Both  the  information  rate  and  channel  capacity  arise  from  the  following 
fundamental  expression  for  the  information  rate.  The  expressions  here  are  for  a  single  sided  bandwidth 
B. 


(i.i) 


H  is  the  average  information  rate  in  bits  for  each  pixel  (with  the  base  2  logarithm).  Stgt(^q)  is  the  power 
spectral  density  of  the  region  of  interest  or  target  as  displayed  (or  in  the  image  plane)  including  noise 
and  sensor  degradations.  N({,q)  is  the  power  spectral  density  of  the  "noise."  Noise  represents  the 
threshold  level  of  the  signal  and  can  be  treated  as  this  more  general  term.  For  example,  if  the  above 
equation  is  applied  with  human  observers,  the  denominator  term  might  become  the  contrast  threshold 
function  rather  than  just  detector  or  system  noise.  £,r)  are  the  spatial  frequency  variables  in  appropriate 
units  such  as  cycles  per  pixel.  B  represents  the  two  dimensional  bandwidth.  For  digital  imagery  the 
bandwidth  is  limited  by  the  half  sample  frequency  (0.5  cycles/pixel  in  each  dimension.)  Note  that  for 
separable  functions  (horizontal  and  vertical  directions)  the  total  information  rate  is  the  sum  of  the  rates 
for  each  dimension. 

The  channel  capacity  results  from  the  information  rate  under  ideal  assumptions.  These  assumptions  are 
never  met  by  real  images  since  the  target  spectrum  is  assumed  to  be  constant  across  the  bandwidth. 
Such  a  spectrum  is  white  noise  to  an  observer  with  each  pixel  being  uncorrelated  with  the  adjacent 
pixels.  A  white  spectrum  might  be  approached  by  an  ensemble  of  images  or  targets.  S  is  assumed  to 
consist  of  a  signal  spectrum  with  additive  noise  and  no  sensor  degradation.  Under  the  channel  capacity 
assumptions,  the  upper  bound  of  H  becomes  the  following. 

C  =  B\og(\+SNR2)  (1.2) 

C  is  the  channel  capacity  and  SNR  is  the  signal  to  noise  ratio.  The  square  of  SNR  is  the  power  signal  to 
noise  ratio.  In  this  case  the  term  signal  does  not  include  the  additive  noise.  The  one  accounts  for  the 
additive  noise.  This  equation  implies  the  physical  information  limit  for  the  system  is  the  number  of 
resolvable  levels  of  signal  (signal  power  divided  by  noise  power)  times  the  bandwidth  of  the  channel. 

The  encoding  of  the  information  is  ideal  and  C  is  in  bits/pixel. 

For  the  binary  output  beam  break  sensor  it  is  convenient  to  think  of  locating  an  edge  in  one  dimension. 
The  sensor  output  encodes  the  time  that  the  edge  passes  the  detector.  From  an  information  standpoint 
this  action  is  equivalent  to  locating  the  edge  in  space.  Refer  to  Figure  2  to  translate  location  into 
detector  output.  The  threshold  results  in  digital  output  with  an  edge  at  the  optical  axis,  but  that 
location  is  made  uncertain  by  the  noise.  Since  the  detector  binary  output  toggles  at  the  midpoint  of  the 
analog  edge  response,  the  signal  level  corresponds  to  half  of  the  beam  width.  The  channel  capacity 
represents  the  limit  to  which  we  can  know  the  location  within  the  analog  edge  response.  Keep  in  mind 
that  the  channel  capacity  assumes  a  white  spectrum  which  is  equivalent  to  locating  a  multitude  of  edges 
within  the  analog  edge  response.  No  location  information  results  when  the  detector  output  remains 
fixed  in  either  state,  blocked  or  unblocked.  Edge  location  information  only  results  from  a  transition. 
Channel  capacity  then  represents  the  number  of  bits  required  to  encode  the  maximum  number  of 
different  positions  within  the  analog  edge  response  that  the  sensor  can  isolate  based  on  signal  level  and 
noise  level. 

Information  Rate  Model  for  Binary  Output  Beam  Break  Sensor 

The  information  theoretic  model  is  based  on  Eqn.  1.  As  suggested  above  we  choose  to  represent  the 
horizontal  direction  as  spatial  pixels.  The  target  speed  times  the  sample  period  determines  the  size  of 


the  pixels.  Therefore  Eq.  1  is  directly  applicable  to  our  beam  break  sensor.  The  linear  systems  model 
requires  some  assumptions  regarding  the  threshold  process  which  is  not  linear.  For  our  region  of 
interest,  we  choose  the  target  itself  with  only  enough  background  pixels  to  model  the  transitions  from 
unblocked  to  blocked.  Our  target  is  a  rectangular  block  that  moves  in  the  horizontal  direction  only. 
While  an  edge  target  is  interesting,  it  results  in  a  spectrum  that  violates  assumptions  inherent  in  Eq.  1, 
namely  having  a  form  that  has  a  defined  logarithm.  One  could  blurthe  ideal  edge,  but  the  block  shape 
more  nearly  matches  the  typical  applications  of  classification  based  on  a  profile.  With  these  thoughts  in 
mind,  the  information  rate  metric  for  the  horizontal  direction  is  given  by  the  following. 


(1.3) 


The  limits  on  the  integral  are  in  cycles  per  pixel  where  the  upper  limit  is  the  half  sample  frequency. 
Consequently,  a  similar  expression  is  also  used  for  Hy ,  the  information  rate  metric  for  the  vertical 
direction.  The  separable  system  assumption  means  the  total  information  rate  is  the  sum  of  the  two 
directional  rates.  In  practice  the  pixel  size  is  rarely  the  same  in  the  two  directions,  but  the  limits  are  the 
same  in  cycles  per  pixel.  TGT is  the  target  spectrum  and  the  square  represents  the  power  density.  For 
an  ideal  block  target,  the  spectrum  is  the  familiar  sine  function.  The  same  convention  is  used  for  the 
noise  spectrum  N  and  MTF  represents  the  sensor  degradations  such  as  blur. 

The  threshold  process  model  is  incorporated  through  the  following  approach.  Figure  2  represents  the 
analog  output  edge  response  and  shows  the  typical  blur.  However,  the  digital  output  with  a  threshold 
at  fifty  per  cent  blocked  is  a  sudden  transition,  say  from  0  unblocked  to  1  for  blocked  condition.  As  a 
result  the  blur  does  not  impact  the  digital  output.  Therefore,  if  one  is  using  an  automated  classification 
algorithm,  the  sensor  degradation  function  is  set  to  one  across  the  spectrum.  If  one  converts  the  digital 
output  to  analog,  a  reconstruction  blur  occurs.  But  the  blur  is  mostly  within  a  single  sample  period 
except  for  some  ringing.  Its  apparent  from  the  example  system  discussed  below  that  the  threshold 
process  significantly  reduces  the  effect  of  blur,  even  for  analog  outputs. 


Example  Beam  Break  Sensor 

Eqn.  3  provides  a  general  model  for  determining  the  information  rate  metric  for  a  binary  output  beam 
break  sensor.  This  section  applies  Eqn.  3  to  an  example  system.  While  the  chosen  system  is  typical,  the 
ensemble  of  possible  sensors,  detector  transmitter  separation,  etc.  is  large.  However,  the  purpose  of 
illustrating  the  process  is  well  served  by  a  numerical  example. 

Experiments  to  determine  sensor  parameters  include  a  repeated  trials  approach.  An  edge  target  (tall, 
wide  block)  is  moved  through  the  sensor  repeatedly  and  the  position  that  changes  the  binary  state  of 
the  sensor  output  is  recorded.  The  standard  deviation  of  the  edge  location  that  toggles  the  sensor  is  the 
edge  location  uncertainty.  For  vertical  uncertainty  the  vertical  location  of  the  block  is  varied.  The 
variance  of  the  edge  location  that  toggles  an  adjacent  detector  is  the  vertical  edge  location  uncertainty. 


Actual  measurements  on  a  garage  door  safety  sensor  confirm  the  advantages  of  a  binary  output  device 
in  reducing  edge  location  uncertainty.  This  particular  sensor  uses  1.8  cm  diameter  apertures  for 
transmitter  and  detector/receiver,  which  are  spaced  2.7  m  apart.  The  threshold  level  is  approximately 
50%,  an  optimum  choice  for  detection  at  the  optical  axis  independent  of  target  location  between  the 
transmitter  and  receiver.  Both  the  field  of  view  of  the  detector  and  the  beam  width  from  the 
transmitter  are  approximately  1  m  full  width  at  2.7  m  distance.  The  uncertainty  in  edge  location  is  0.5 
mm  (±  0.2  mm).  The  symmetry  of  the  device  means  this  uncertainty  is  the  same  for  both  horizontal  and 
vertical  directions.  The  edge  location  lies  on  the  optical  axis,  within  the  uncertainty  of  the  determining 
the  position  of  the  optical  axis  (±  1  mm). 

Additional  uncertainty  arises  from  the  temporal  sampling  of  a  moving  target  (horizontal)  and  the  spatial 
sampling  from  using  an  array  of  separate  detectors  (vertical.)  For  the  horizontal  direction  assume  a  1 
m/sec  speed.  This  is  a  reasonable  speed  for  walking  and  it  is  also  easy  to  scale  to  different  speeds. 
Sampling  at  one  thousand  samples  per  second  yields  an  equivalent  spatial  pixel  size  of  1  mm.  Assuming 
a  uniform  distribution  of  edge  location  about  any  given  sample,  the  variance  is  then  1  mm  squared 
divided  by  twelve.  The  variances  add  since  sampling  uncertainty  is  independent  of  the  edge  location 
variance. 

Sampling  in  the  vertical  direction  is  accomplished  by  different  sensors,  so  pixel  size  is  the  physical 
distance  between  the  sensors.  We  assume  a  typical  value  of  100  mm  as  the  sample  spacing.  Therefore, 
the  variance  is  100  cm2  divided  by  twelve.  The  sampling  uncertainty  in  the  vertical  direction  is  also 
independent  of  the  edge  location  uncertainty.  The  variances  add,  but  the  sampling  uncertainty  is  the 
dominant  uncertainty  for  the  vertical  direction. 

Eqn.  2  for  the  channel  capacity  can  now  be  used  for  the  example  sensor.  We  assume  a  linear 
relationship  between  location  of  the  edge  and  the  analog  detector  output  (see  Figure  2).  One  can  then 
transform  edge  location  into  fractional  signal  level.  In  Eqns.  2  and  3  all  powers  occur  in  ratios. 

Therefore,  fractional  signal  level  (edge  location)  is  sufficient  information  for  calculating  channel  capacity 
and  realizable  information  rates.  The  beam  size/detector  field  of  view  at  the  midpoint  location  is  500 
mm.  The  detector  output  toggles  at  with  the  edge  at  250  mm  from  the  beam  edge,  that  is,  it  toggles  at 
optical  axis.  Therefore  the  signal  level  is  half  of  the  unblocked  radiation  level,  or  0.5  fractional  signal 
level.  The  noise  is  the  standard  deviation  of  the  measured  values  from  the  previous  paragraphs.  For  the 
horizontal  direction,  the  noise  is  0.315  mm  (6.3  x  10'4  fractional  signal).  Eqn.  2  then  yields  9.6  bits  per 
pixel  for  the  channel  capacity. 

The  channel  capacity  for  the  vertical  direction  is  obtained  from  the  same  signal  level,  but  an  increased 
noise  from  the  larger  sample  spacing.  The  signal  level  (in  position  units)  is  250  mm  while  the  noise  is 
28.9  mm  (100  mm  detector  separation).  This  yields  a  channel  capacity  of  3.1  bits  per  pixel.  With  a 
separable  model  the  two  add  for  a  total  channel  capacity  of  12.7  bits  per  pixel. 

Eqn.  3  for  the  realizable  rate  is  applied  through  selecting  a  target  and  including  the  sensor  degradation  if 
appropriate.  The  channel  capacity  is  based  on  a  white  spectrum  which  is  unrealistic  for  a  profiling 
sensor.  Consequently,  we  choose  a  block  with  typical  size  (700  mm  wide  by  1400  mm  high)  for 


human/other  animals  to  illustrate  the  model  and  obtain  a  typical  realizable  rate.  We  first  consider  a 
digital  output  and  set  the  MTF  to  1  for  all  spatial  frequencies.  The  integration  indicated  in  Eqn.  3  is 
carried  out  numerically  and  results  in  a  realizable  rate  of  0.9  bits  per  pixel  for  the  horizontal  direction 
and  0.5  bits  per  pixel  for  the  vertical  direction. 

Sensor  degradations  are  included  through  the  MTF  function.  To  illustrate  we  now  consider  an  analog 
output  and  use  a  reconstruction  blur  for  the  degradation.  The  blur  is  assumed  to  be  only  between  the 
digital  edge  (pixel  value  1)  and  the  nearest  off  pixel  (value  0.)  The  fidelity  of  the  reconstruction 
improves  only  slightly  as  the  number  of  pixels  on  the  target  increase  from  a  few  to  several  hundred. 

With  our  numerical  assumptions  we  have  500  pixels  on  the  target  in  the  horizontal  direction  and  14 
samples  in  the  vertical  direction.  We  use  the  following  Gaussian  function  fit  to  the  reconstructed  analog 
signal  between  the  two  pixels  defining  the  edge. 

MIFX  (£)  =  exp  (-2^lunf )  (1.4) 

The  notation  is  for  the  horizontal  direction  and  a  similar  function  is  used  for  the  vertical  direction.  The 
fit  determines  the  best  value  of  ab|Urx.  To  within  the  accuracy  of  measured  values,  the  best  fit  is  the  pixel 
separation  divided  by  2.5,  for  either  vertical  or  horizontal  directions.  Figures  4  and  5  illustrate  the  signal 
spectrum  and  the  MTF2  for  the  horizontal  and  vertical  directions  respectively.  The  target  plus  noise 
spectra  have  no  zeros  at  the  minima,  seen  more  clearly  in  Figure  5.  Applying  the  degradation  does 
reduce  the  SNR  below  1  near  some  of  the  minima.  Those  regions  are  excluded  from  the  integration 
since  the  logarithm  is  undefined  there.  The  resulting  realizable  rates  with  this  reconstruction  blur 
degradation  are  0.7  bits  per  pixel  in  the  horizontal  direction  and  0.4  bits  per  pixel  in  the  vertical 
direction. 

The  average  information  rate  coupled  with  the  number  of  pixels  on  the  target  provides  an  estimate  of 
the  information  content.  The  minimum  number  of  pixels  for  successful  classification  will  be  discussed 
below. 


Additional  Results  of  Interest  from  the  Information  Theoretic  Model 

Quantifying  the  value  of  information  might  be  accomplished  by  the  process  described  next.  Consider  a 
two  sensor  system  that  includes  a  highly  capable  intelligence/surveillance/reconnaissance  (ISR)  imager. 
The  second  sensor  is  much  more  limited,  perhaps  a  profiling  sensor  as  modeled  above.  We  choose  to 
think  of  the  second  sensor  as  augmenting  the  information  from  the  ISR  imager  and  we  attempt  to 
quantify  the  value  of  the  additional  information. 

Our  notional  ISR  sensor  has  one  million  pixels  and  a  ground  sample  distance  of  0.5  meters.  A  single 
static  image  can  be  analyzed  for  the  average  information  rate  within  a  region  of  interest  that  includes 
the  profiling  sensor.  With  just  this  single  static  image,  it  is  difficult  to  even  locate  a  human  in  the  image. 
However,  suppose  the  profiling  sensor  has  classified  an  object  as  a  human  just  prior  to  the  acquisition  of 
the  ISR  image.  This  additional  information  perhaps  enables  one  to  classify  a  blob  of  pixels  as  a  human. 


The  information  theoretic  model  presented  here  estimates  the  information  rate  for  a  profiling  sensor. 
Such  sensors  have  provided  excellent  classification  accuracy  in  hundreds  of  examples  of  human/animal 
data.  Consequently,  the  data  rate  coupled  with  the  number  of  pixels  required  provides  an  estimate  of 
the  information  value  of  the  added  classification  result  from  the  profiling  sensor.  This  increased 
information  could  be  also  be  modeled  as  an  increased  ISR  resolution,  perhaps  an  equivalent  ground 
sample  distance  of  0.1  meters  within  that  small  region  of  interest.  This  approach  to  establishing  the 
value  of  information  can  be  further  developed,  as  discussed  in  the  conclusion  as  a  future  effort.  Of 
course  if  the  ISR  sensor  video  stream  is  available,  one  can  detect  a  moving  object  the  size  of  a  human 
without  additional  sensors.  However,  the  typical  ISR  sensor  view  point  is  from  above,  making 
human/other  classification  somewhat  harder.  The  example  is  chosen  to  illustrate  the  value  of  added 
information  from  additional  sensors,  not  to  discuss  sensor  selection. 


The  information  rate  from  the  information  theoretic  model  is  given  as  an  average  per  pixel,  so  one  must 
consider  how  many  pixels  are  required  to  execute  a  task  such  as  classification.  One  approach  to 
establishing  the  number  of  pixels  required  for  classification  is  to  use  uncertainty  analysis.  The  primary 
feature  for  human/animal  classification  is  the  height  to  width  ratio.  This  means  the  number  of  pixels 
required  can  be  estimated  from  the  required  accuracy  of  measuring  the  height  and  width.  One  uses  the 
same  "noise"  estimates  to  get  the  standard  deviation  used  in  uncertainty  discussions.  Our  example 
system  will  be  used  for  the  numerical  results.  Our  goal  is  uncertainty  of  the  ratio  of  height  to  width,  not 
the  individual  parameters  of  height  and  width. 


In  equation  form,  the  following  equation  describes  the  uncertainty  for  the  height  to  width  ratio.  First 
order  uncertainty  of  the  ratio  adds  the  fractional  uncertainties  of  the  height  and  width. 
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h  and  w  are  height  and  width,  A h  and  Aw  are  the  uncertainties,  and  H.O.T.  denotes  terms  of  second 
order  and  higher  in  the  fractional  uncertainties  A h/h  and  A w/w.  A  minus  sign  in  front  of  the  width 
fractional  uncertainty  has  been  replaced  by  a  plus  sign,  appropriate  for  combining  uncertainties.  Two 
edges  must  be  located  to  measure  height  and  width  although  the  ground  might  form  one  edge  for 
height.  For  numerical  purposes  we  assume  the  uncertainty  for  each  edge  location  measurement  is 
independent  of  the  uncertainty  for  any  other  edge  location  measurement. 


Our  example  system  provides  an  abundance  of  samples  in  the  horizontal  direction  so  the  uncertainty  is 
dominated  by  the  vertical  uncertainty  where  Ah  is  40.8  mm  (two  edges).  For  some  insight  the  minimum 
size  required  to  block  50%  of  the  radiation  is  somewhat  less  than  150  mm  (136  mm  by  analysis.)  Four 
pixels  (400  mm)  of  height  reduces  the  vertical  fractional  uncertainty  to  approximately  0.1  or  10%.  The 
minimum  number  of  pixels  in  the  horizontal  direction  is  overdesigned  in  the  example  system.  Therefore 
for  estimating  the  minimum  number  of  pixels,  we  create  a  notional  system  with  a  horizontal  pixel 
spacing  of  100  mm.  This  redesigned  pixel  size  will  have  the  same  horizontal  uncertainty  as  the  vertical 


direction.  So  it  is  suggested  that  one  use  four  pixels  minimum,  corresponding  to  a  400  mm  target  size. 
Such  a  design  accommodates  height  to  width  ratios  with  a  maximum  of  4  and  a  minimum  of  V* .  This  4  x 
4  pattern  yields  16  pixels  total  and  a  reasonable  uncertainty.  Few  moving  objects  are  square,  so  fewer 
than  16  pixels  might  suffice.  A  3x3  pattern  is  obtained  using  Johnson's  criteria  for  recognition  [Johnson, 
1958]  with  Holst's  modification  for  low  clutter  [Holst,  1995],  The  typical  sized  block  target  used  in  this 
effort  produces  14  x  7  pixels  with  100  mm  horizontal  pixels  or  approximately  100  pixels.  The  system 
used  for  data  collections  is  intentionally  overdesigned  to  accomplish  studies  that  classify  more  than  two 
classes,  e.g.  humans,  animals,  humans  with  backpacks  or  humans  with  weapons. 

These  considerations,  uncertainty  analysis  and  minimum  target  sizes  that  are  detected,  suggest  that  16 
pixels  or  fewer  may  be  sufficient.  However,  the  pixels  must  be  sized  such  that  approximately  4  pixels 
are  across  each  dimension  of  the  target.  The  edges  are  the  significant  feature.  Adding  many  more 
pixels  that  are  highly  correlated  does  not  add  much  information.  The  number  of  pixels  on  target  has  a 
slightly  different  connotation  with  profiling  sensors  as  contrasted  with  imaging  sensors.  More  pixels  in 
an  imaging  system  may  provide  useful  intra-target  information.  In  a  profile  there  is  essentially  no  intra¬ 
target  information.  Obviously  more  pixels  and  smaller  pixels  can  provide  more  detail  in  the  outline  of 
the  target,  even  in  the  profiling  sensor  [Chari,  2010]. 

Conclusion  and  Future  Effort 

This  paper  describes  a  process  for  obtaining  the  information  rate  per  pixel  of  a  profiling  sensor.  In 
effect,  a  profiling  sensor  provides  information  at  transitions.  For  example,  the  binary  output  beam 
break  sensor  makes  a  transition  from  0  to  1  or  from  1  to  0.  The  location  of  these  transitions  represents 
the  information  content  so  a  finer  sampling  produces  more  information,  but  the  sampling  is  limited  by 
the  noise  level.  A  block  target  is  used  to  illustrate  the  process  and  numerical  results  are  given. 

One  motivation  for  developing  the  information  theoretic  model  is  to  enable  the  optimization  of  sensor 
design  within  a  system  of  sensors.  One  approach  to  the  optimization  is  suggested  by  the  convex 
optimization  technique  [Joshi,  2009],  One  considers  a  new  design  to  simply  be  an  additional  sensor 
available  to  be  "selected"  for  use  by  the  system.  Variations  of  a  particular  parameter  create  new 
sensors  that  are  possibly  available.  The  optimization  technique  then  chooses  from  all  the  possible 
designs  being  considered  as  a  sensor  selection  process.  Information  rate  is  envisioned  as  the  metric  that 
ties  all  of  the  sensors  together  in  a  system.  This  enables  one  to  "design"  a  particular  sensorso  that  the 
system  level  performance  is  optimum,  at  least  by  some  chosen  criterion. 

Future  efforts  include  expanding  the  model  to  include  passive  profiling  sensors.  This  category  includes 
different  approaches,  one  of  which  is  In  effect  an  image  processing  technique.  Consequently,  the 
information  throughput  for  an  image  processing  algorithm  can  result  from  the  modeling  effort.  Other 
approaches  utilize  linear  arrays.  As  shown  here  the  moving  targets  are  scanning  themselves  when  they 
pass  through  the  field  of  view  of  the  linear  array.  Therefore,  the  process  presented  in  this  paper  should 
be  directly  applicable  to  this  particular  type  of  passive  sensor. 

In  some  sense  the  numerical  results  coupled  with  classification  algorithm  performance  may  provide  an 
estimate  of  the  value  of  information  for  this  classification  task.  This  points  the  way  to  creating  a  table  of 


tasks  with  the  estimated  value  of  information  associated  with  successful  completion  of  the  task.  One 
could  use  the  table  to  estimate  the  value  of  information  not  directly  in  the  table.  The  user  either  finds 
an  equivalent  task  in  the  table  or  "brackets"  the  task  by  finding  a  task  slightly  easier  and  another  slightly 
harder  from  the  table.  The  NIIRS  metric  is  often  structured  this  way.  One  could  develop  such  a  table  by 
utilizing  human  perception  studies,  both  existing  results  and  new  ones  conducted  for  this  purpose. 
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Figurel  A  top  view  depicts  the  detector  field  ofview  and  the  transmitter  beam  width 
for  a  beam  break  sensor.  The  arrows  represent  edges  being  moved  into  the  field  of  view.  At 
position  1  the  detector  field  ofview  limits  the  edge  detection.  At  position  3  the  transmitter 
beam  width  is  the  limit.  It  is  apparent  the  amount  of  edge  movement  required  for  a  detector 
output  change  depends  on  the  edge  location  between  the  detector  and  the  transmitter. 


Figure  2  This  figure  simulates  the  analog  detector  response  to  an  edge  moving  across 
the  field  of  view  with  added  noise.  It  demonstrates  that  edge  position  within  the  field  of 
view  has  a  one-to-one  relationship  with  fractional  optical  power,  except  for  noise. 


Human  with  backpack  Pony  Human  without  backpack 


Figure  3  Output  from  the  binary  output  beam  bread  sensor  is  displaced  in  a  two 
dimensional  format.  Hundreds  of  these  examples  have  been  collected  for  testing  profiling 
sensors  and  classification  algorithms  for  processing  their  output  data. 
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Figure  4  This  figure  illustrates  the  example  system  quantities  used  to  find  a  numerical  value 
for  the  information  rate.  The  target  plus  noise  power  spectrum  is  normalized  to  one  for 
illustration  (actual  peak  is  2502  mm2.)  The  horizontal  axis  stops  at  0.005  cy/mm  whereas 
the  integration  goes  to  0.5  cy/mm.  The  horizontal  scale  as  chosen  matches  that  for  the 
vertical  direction  shown  in  Figure  5,  The  solid  blue  curve  is  the  MTF2  and  the  dotted  red 
curve  shows  the  target  spectrum  normalized  to  a  peak  value  of  1. 


Power  Spectrum  of  Target  +  Noise  And  MTF2 


Figure  5  The  solid  blue  curve  is  the  square  of  the  MTF  and  the  dotted  red  curve  is  the 
target  power  plus  the  noise  power.  For  this  vertical  direction  the  integration  ends  at  5xl0*3 
cycles  per  millimeter. 


Equation  Chapter  5  Section  SPerformance  Model  for  Micro-resonator 
Sensors  Leading  to  Information  Rate 

A  common  feature  to  many  micro  resonator  based  sensors  is  a  sphere  being  deformed.  That 
deformation  changes  the  resonant  wavelength.  The  resulting  change  in  resonant  wavelength  is  then 
detected.  The  detection  is  accomplished  by  noting  the  resonant  wavelength  shift  from  before  and  after 
the  deformation.  The  resonance  wavelength  is  the  point  of  greatest  loss  from  the  transmitted  beam. 

The  relationship  between  the  deformation  and  the  individual  "force"  being  measured  must  be  modeled. 
Here  "force"  means  field  quantity  being  sensed  as  well  as  force.  This  elasticity  model  depends  on 
materials  and  the  resonator  geometry.  Much  of  this  elasticity  modeling  is  already  available  and  is 
summarized  well  in  [1],  Their  model  relates  the  force  being  measured  to  the  change  in  wavelength  of 
the  resonant  cavity.  Essentially  this  is  the  model  of  the  common  feature  to  those  sensors  of  interest. 

The  model  presented  in  [1]  does  not  treat  the  statistical  aspects  of  the  sensor.  For  example,  what  are 
the  noise  sources  in  laser  source  including  its  current  driver?  How  much  wavelength  jitter  can  be 
expected  from  the  laser  diode?  Similarly,  we  need  to  model  the  detector  noise. 

We  want  to  also  to  model  the  digitization  issues  of  bit  depth  and  dynamic  range  and  sampling  noise. 

The  actual  signal  processing  reported  in  [1]  is  a  correlation  of  the  model  pattern  before  the  force  and 
after  the  force.  The  correlation  will  reveal  the  shift  and  this  shift  can  be  expressed  as  a  wavelength  shift. 
The  model  as  presented  in  [1]  is  an  analog  model. 

Shapes  other  than  spherical  have  been  studied  for  potential  use  in  micro-resonator  sensors.  Perhaps 
one  can  be  creative  with  the  basic  spherical  shape.  Is  there  an  optimal  shape  that  might  even  be 
different  for  different  sensing  modalities? 

The  model  in  [1]  uses  a  first  order  ray  model  for  the  resonance.  A  more  sophisticated  (complicated) 
model  would  use  the  electromagnetic  fields  inside  the  resonator  so  that  any  mode  effects  on  the 
resonance  are  included.  The  em  fields  inside  the  cavity  for  some  cavity  shapes  are  available  in  the 
literature  [2].  For  example,  prolate  (and  oblate)  spheroidal  shapes  are  two  important  cases  that  are 
already  solved  [2],  However,  the  first  order  ray  model  appears  to  be  very  successful.  This  is  likely  due  to 
the  fact  that  the  spheres  being  used  are  much  larger  than  the  resonant  wavelength.  If  future  work 
involves  smaller  spheres,  the  ray  model  might  not  be  as  successful.  The  smallest  mode  volume 
theoretically  possible  is  on  the  order  of  A3  [3]. 

To  summarize,  the  performance  model  for  microresonator  sensors  has  three  parts,  the  elasticity  model 
relating  force  to  deformation,  the  resonance  model  relating  deformation  to  wavelength  change,  and  the 
signal  processing  and  noise  model. 


5.1  Elasticity  Model 

loppolo  et  al.  summarize  the  elasticity  of  the  sphere  and  its  response  to  a  force  from  rigid  plates  that  are 
diametrically  opposite(l].  The  solutions  they  reference  are  for  a  fairly  rigid  material  and/or  a  small  force 
and  do  not  apply  where  the  plates  touch  the  sphere  (squashed  flat  region).  In  loppolo's  coordinate 
system  (typical  spherical  coordinates),  there  is  symmetry  with  respect  to  c|>  and  variation  in  the  polar 
angle  0  and  the  radius  r.  What  is  varying  is  the  displacement  u  of  a  given  point  within  the  sphere  and 
the  pressure  plates  are  at  the  poles,  0=0  and  0  =  k. 

They  begin  with  the  Navier  equation  for  the  displacement  (strain)  of  a  given  point. 

V2w  +  — - — VV'U  =  0  (5.1) 

\-2v 

v  is  the  Poisson  ratio.  The  Poisson  ratio  relates  stress  and  strain  in  directions  perpendicular  to  the  force. 
Squashing  at  the  poles  causes  bulging  at  the  equator,  0  =  n/2,  and  the  Poisson  ratio  models  that  effect. 

With  the  azimuthal  symmetry  the  solution  is  given  by 

Ur  =  Z  [A»  (»  +  1 )  (n  ~  2  +  4 V )  r”+'  +  Bn”rn'X ]P»  ( COS  6)  (5.2) 
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An  and  Bn  are  determined  from  boundary  conditions,  but  will  be  related  to  another  constant,  Hn,  before 
applying  the  conditions.  Pn(cos&)  denote  the  Legendre  polynomials  of  order  n. 

The  resonance  condition  for  the  micro  resonator  is  that  a  phase  shift  of  an  integral  multiple  of  2k  occur 
within  the  resonator.  The  authors  use  the  ray  model  and  the  assumption  that  the  circumference  of  the 
sphere  is  the  path.  That  leads  to  the  following  equation. 

27lnQa  =  kX  (5.3) 

a  is  the  radius  of  the  sphere  and  A  is  the  freespace  wavelength.  Since  we  are  interested  in  shifts  in  the 
resonant  wavelength,  the  authors  take  the  differential  change  and  write  the  following. 

dX  dnn  da  ,  , 
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The  change  in  the  index  of  refraction  is  a  result  of  a  force  (stress)  at  the  point  being  considered. 
Consequently,  the  strain  result  and  the  shear  modulus  of  the  material  are  used  to  obtain  the  stress. 
Eqn.(5.5)  shows  only  the  rr  component.  The  other  components  are  not  relevant  under  the  assumptions 
for  this  analysis. 
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By  considering  the  interface  between  the  plates  applying  the  force  and  the  sphere  itself,  there  is  no 
"rolling"  of  the  material.  This  means  the  boundary  conditions  are  as  follows. 
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0O  is  the  greatest  value  of  0  for  which  the  contacting  plate  touches  the  sphere.  Similarly,  a0  is  the  radius 
of  the  contact  area  between  the  plates  and  the  sphere,  that  is,  from  the  polar  axis  to  point  on  the 
surface  of  the  sphere  where  the  pressure  plate  contact  ends.  The  expression  for  a0  follows. 
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E  is  Young's  modulus  for  the  material  and  a  is  the  radius  of  the  sphere.  The  expression  for  pressure 
distribution  under  the  contacting  plate  is  given  by  the  following. 

p{p)  =  si°2  (0)  <5-8) 

l7ta0 

The  symmetry  of  this  arrangement  yields  a  deformation  that  does  not  vary  with  the  azimuthal  angle  cj) 
and  is  an  even  function  with  respect  to  0  (about  0  equal  to  n/2).  As  a  result  the  expansion  of  Eqn  (5.9) 
uses  only  the  even  orders  of  the  Legendre  polynomials. 

Ma)=Stfnpn(cos^)  (5-9) 

n=0 

The  coefficients  Hn  are  found  by  invoking  the  orthogonality  of  the  Legendre  polynomials.  The  result 
follows. 


Hn  =(4;i+I)  [  Jal  - a2sin2(8)Pn  (cos#)sin  ddd  (5.10) 
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The  integral  is  evaluated  numerically  for  a  specific  example.  Some  details  of  the  numerical  evaluation 
are  presented  in  Appendix  A  for  this  chapter. 

Once  the  set  of  Hn  is  obtained,  the  An  and  Bn  in  Eqn  (5.2)  can  be  found.  The  following  expressions  for  An 
and  Bn  are  obtained  by  using  Eqn  (5.9)  and  Eqn(5.5)  to  satisfy  the  boundary  conditions  in  Eqns  (5.6), 

(5.7)  and  (5.8). 
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The  a  n  factor  in  both  An  and  Bn  is  associated  with  the  rn  factor  in  Eqn  (5.5)  to  avoid  a  numerical  disaster 
as  n  increases.  For  the  cases  of  interest,  a  is  on  the  order  of  hundreds  of  micrometers.  Therefore,  as  a  is 
raised  to  higher  and  higher  powers  of  n,  the  values  of  An  and  Bn  grow  larger  and  larger. 

5.2  Signal  Processing  and  Noise  Model 

The  feature  extracted  from  the  signal  is  the  resonant  wavelength  shift  caused  by  a  force.  The  shift  is 
obtained  by  processing  the  output  from  the  photodetector.  That  output  shows  more  than  a  single 
mode,  so  there  is  a  pattern  of  minima  in  the  detector  output.  The  ramp  in  the  output  due  to  the  current 
ramp  is  removed  from  the  data.  The  entire  pattern  shifts  as  a  force  is  applied  to  the  resonator. 
Correlation  is  used  to  measure  the  amount  the  pattern  shifts  in  response  to  the  force.  Most  published 
results  use  a  large  signal  so  that  signal  processing  uncertainty  and  noise  is  not  likely  significant.  For  field 
applications,  however,  minimizing  power  may  result  in  operating  with  a  low  signal-to-noise  ratio. 

Hence,  the  current  section  models  the  major  contributors  to  the  uncertainty  in  the  data  output. 


Actually,  wavelength  is  not  measured,  but  rather  a  current  level  from  a  ramp  used  to  drive  the  laser 
diode.  Expressing  the  shift  in  wavelength  is  the  traditional  way  of  expressing  the  change.  As  a  result  of 
this  choice,  temporal  noise  (in  the  laser  driver  and  in  the  photodetector)  is  converted  to  wavelength 
through  the  calibration  curve  relating  current  and  wavelength  for  the  laser.  For  illustration  purposes  the 
conversion  factor  of  5  picometers  per  milliamp  will  be  used. 


The  graphic  below  illustrates  the  major  factors  included  in  the  signal  processing  and  noise  model. 
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Figure  5.1  This  overall  view  shows  the  factors  included  in  the  signal  processing  and  noise 
model. 

For  combining  noise  sources,  the  independence  of  the  various  sources  of  noise  allows  us  to  add  the 
individual  fractional  variances.  Or,  if  each  noise  is  converted  to  the  same  units  (such  as  nm),  the 
variances  may  be  added.  The  overall  standard  deviation  is  then  the  square  root  of  the  variances.  In  the 
example  below,  all  noise  terms  are  converted  to  equivalent  wavelength.  Eqns  (5.13),  (5.14)  and  (5.15) 
summarize  the  noise  and  uncertainty  in  the  signal  processing  and  noise  model.  Other  terms  can  be 
added  and  some  of  these  can  be  deleted  as  the  model  is  applied  to  specific  cases. 

laserNoiseVar  =  powerNoiseVar  +  freqNoiseVar  + 

driverNoiseVar  +  sourceSamplingNoiseVar 

detSigProcVar  =  detTempNoiseVar  +  sampleNoiseVar 
+dipSeparationProcessingVar 

var  =  laserNoise  Var  +  detSigProcVar 

This  model  implicitly  assumes  that  the  signal  processing  takes  place  at  the  sensor.  That  means  the 
output  of  the  signal  processing  step  is  the  end  of  the  model.  There  is  not  a  transmission  model  for  the 
noise  and  corruption  that  occurs  in  transmitting  the  data.  Other  modes  of  operation  such  as 
transmitting  the  detector  output  are  possible.  The  model  can  be  adapted  to  end  the  sensor  part  of  the 
model  at  the  appropriate  step.  One  may  need  to  add  a  transmission  model. 


(5.13) 

(5.14) 

(5.15) 


The  divisions  in  the  model  are  arbitrary.  The  dip  separation  is  the  feature  extracted  from  the  sensor.  It 
is  the  shift  in  wavelength  of  the  dips  or  minima  in  the  power  measured  by  the  detector. 

5.3  Sources  of  Noise  and  Uncertainty 

A  brief  summary  of  the  signals  involved  helps  identify  the  sources  of  uncertainty  in  the  microresonator 
sensor.  A  ramp  of  current  is  used  to  drive  the  laser  diode  and  subsequent  signals  are  compared  in  time 
to  the  ramp.  A  calibration  curve  converts  the  current  values  to  wavelengths.  For  this  diode,  the 
temperature  is  set  by  a  thermoelectric  cooler  since  temperature  changes  result  in  wavelength  changes 
even  without  any  change  in  current.  The  microresonator  changes  its  shape  in  response  to  a  force,  which 
causes  a  shift  in  the  resonant  wavelength.  These  resonances  are  detected  as  losses  in  signal  strength 
from  the  photodiode.  By  noting  where  the  resonances  occur  in  the  current  ramp,  one  can  determine 
wavelengths  and  therefore  the  amount  of  the  wavelength  shift. 

Seeking  the  wavelength  shift  entails  two  types  of  uncertainty,  one  being  temporal  noise  and  the  other 
being  frequency  noise  (wavelength  noise).  The  spectral  linewidth  of  the  laser  represents  frequency 
noise.  Temporal  noise  arises  from  both  the  driver  for  the  laser  diode  and  from  the  photodiode.  Since 
we  are  interested  ultimately  in  wavelength,  we  will  convert  temporal  noise  into  an  equivalent  frequency 
noise.  The  calibration  curve  relating  current  to  wavelength  for  a  typical  laser  diode  is  linear.  Therefore, 
the  two  end  points  are  used  to  approximate  the  slope,  AA/Ai.  For  one  laser  diode  this  slope  or  ratio  is  5 
picometers/milliamp. 

Figure  5.2  below  illustrates  the  conversion  of  temporal  noise  into  a  current  equivalent  (or  temporal 
equivalent)  which  can  then  be  converted  to  wavelength  uncertainty.  The  calibration  between  current 
and  wavelength  (for  example,  5  picometers  per  milliamp)  converts  the  current  uncertainty  into  a 
wavelength  uncertainty. 


equivalent 

current 

uncertainty 


uncertainty 


Figure  5.2  This  illustrates  the  conversion  of  temporal  amplitude  noise  into  an  equivalent 
wavelength  uncertainty.  Suppose  a  noise  spike  changes  the  amplitude  instantaneously  at 
time  t0  during  the  ramp  of  current  Since  the  ramp  is  used  to  determine  wavelength;  the 
extremities  of  the  spike  then  yield  an  equivalent  uncertainty  in  current.  If  time  during  the 
ramp  is  used  to  determine  wavelength,  then  the  equivalent  temporal  uncertainty  shown  in 
the  figure  can  be  used. 

In  general  sampling  uncertainty  is  well  modeled  as  a  noise.  The  noise  level  is  the  variance  which  is 
modeled  as  arising  from  a  uniformly  distributed  probability  density  function.  The  width  of  the  density 
function  is  the  least  significant  bit  converted  to  the  quantity  being  measured.  The  Example  Calculations 
section  illustrates  the  process  with  a  numerical  example. 

5.4  Detector  Noise  Model 

Noise  in  detectors  is  a  mature  field  of  study,  with  many  detailed  models  available.  The  application  area 
is  the  major  consideration  in  choosing  a  model  for  reasonable  results.  The  model  often  used  for 
photodiodes  in  fiber  optic  applications  is  chosen  here.  In  essence,  four  physical  phenomena  are 
included  in  the  model.  Major  noise  sources  are  shot  noise  driven  by  the  signal  level  itself  and 
background  power,  dark  current  noise  and  thermal  noise  in  the  load  resistance.  The  fourth 
phenomenon  is  that  the  signal  level  is  often  measured  with  the  noise  included.  Therefore,  a  fraction  of 
the  signal  power  is  added  to  the  noise  terms  to  account  for  that  phenomenon.  The  end  result  is  given  as 
Eqn  (5.16)  below  for  the  SNR.  This  is  considered  to  be  an  analog  signal,  so  sampling  noise  is  not 
included,  or  equivalently,  the  signals  are  well  sampled.  Other  assumptions  are  that  the  noise  spectra 
are  flat  within  the  electronic  bandwidth,  so  integrations  over  the  band  result  in  a  multiplication  by  the 
bandwidth.  This  model  also  requires  modification  for  avalanche  photodiodes. 
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The  notation  is:  is  and  iN  are  signal  and  noise  currents,  rj  is  the  quantum  efficiency  of  photon-to-electron 
conversion,  q  is  the  electron  charge  in  Coulombs,  Ps  is  the  signal  power  in  watts  (possibly  plus  the  noise 
power),  h  is  Planck's  constant,  v  is  the  frequency  of  the  light,  P6  is  the  background  power  that  reaches 
the  detector,  Af  is  the  electronic  bandwidth  of  the  detection  circuit,  k  is  Boltzmann's  constant,  T  is  the 
temperature  in  degrees  Kelvin,  RL  is  the  load  resistance,  and  a  is  the  fraction  of  the  signal  power  within 
the  electronic  frequency  bandwidth  that  results  from  the  noise.  The  first  three  terms  in  the 
denominator  are  the  shot  noise  variance,  the  dark  current  variance  and  the  thermal  noise  variance.  The 
fourth  term  in  the  denominator  is  the  correction  factor  for  the  signal  actually  being  signal  plus  noise. 


Vendors  commonly  quote  id,  the  dark  current.  At  low  signal  power  (perhaps  less  than  10  pW),  the 
thermal  noise  and  the  dark  current  noise  dominate.  Since  thermal  noise  and  dark  current  noise  do  not 
increase  with  power,  the  signal  to  noise  ratio  improves  as  signal  power  is  increased.  At  some  power 
level  (perhaps  ~  10  pW)  the  shot  noise  and/or  the  fourth  term  in  the  denominator  become  significant.  If 
the  shot  noise  is  dominant,  the  SNR  will  increase  with  increasing  signal  power,  but  only  linearly  rather 
than  as  the  square.  If  the  fourth  term  dominates,  the  SNR  does  not  increase  any  further  with  increasing 
signal  power. 


Vendors  often  quote  the  responsivity  in  amps/watt  instead  of  the  quantum  efficiency.  The  responsivity 
is  given  by  the  following. 


hv 

Using  Eqn  (5.17)  in  Eqn  (5.16)  results  in  a  more  practical  expression  that  follows. 
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5.5  Example  Calculations 

As  an  example  of  sampling  noise,  suppose  the  current  is  being  measured  and  the  dynamic  range  is 
required  to  span  20  mA.  A  16  bit  A/D  converter  resolves  the  20  mA  dynamic  range  into  65536  bins. 
Therefore  the  least  significant  bit  corresponds  to  20/65536  mA  or  approximately  0.3  pA.  With  the 
uniform  probability  density  function,  the  variance  is  given  by  (0.3  pA)2/12  or  approximately  7.7  x  10  1SA2, 
ora  standard  deviation  of  0.88  pA.  This  standard  deviation  in  wavelength  terms  is  (5  pm/mA  x  0.88  x  10’ 
mA)  or  0.44  x  10*3  pm.  Another  sampling  process,  a  temporal  one,  takes  place  as  the  current  ramp 
progresses.  The  uncertainty  associated  with  the  temporal  sample  rate  is  discussed  below. 


Specifications  are  not  often  detailed  for  laser  diodes,  so  the  following  numbers  are  selected  for 
illustrating  the  process  of  determining  wavelength  uncertainty.  The  operating  power  level  is  typically  in 
the  mW  to  tens  of  mW  range,  so  we  assume  8  mW  of  laser  output  power.  At  that  level,  the  spectral  or 
frequency  noise  is  quoted  as  10  MHz,  so  we  will  use  0.06  pm  as  the  equivalent  frequency  noise. 
Amplitude  noise  arises  from  sources  such  as  the  laser  diode  itself,  driver  current  noise  and  temperature 
noise  The  Agere  2300  laser  specifications  quote  the  relative  intensity  noise  as  -  145  dB/Hz.  No  separate 
specifications  were  available  for  driver  noise,  but  it  is  small  based  on  the  overall  noise  amount.  Also,  the 
power  output  stability  of  the  laser  itself  was  not  specified  and  that  too  is  small.  Temperature  variations 
will  manifest  themselves  as  a  change  in  wavelength,  indistinguishable  from  a  change  in  current.  The 
laser  diodes  are  usually  cooled  by  a  thermoelectric  cooler.  The  temperature  fluctuations  may  be  slow 
enough  to  amount  to  drift  more  than  noise.  In  any  case  the  small  overall  intensity  noise  suggests  the 
thermoelectric  cooler  holds  the  temperature  steady  without  adding  significant  noise.  Consequently,  for 
illustration  purposes  the  total  intensity  noise  of  the  laser  source  will  be  -  145  dB/Hz. 

A  typical  responsivity  is  0.8  mA/mW  so  the  signal  level  will  be  6.4  mA. 

The  background  power  reaching  the  detector  is  assumed  to  be  negligible  (PB  =  0)  as  will  the  signal 
fraction  resulting  from  noise  (a  =  0). 

Thermal  noise  current  squared  with  T  =  300  deg  K  is  1.66  x  1C)*20  amps  squared  ohms  per  Hertz.  So  this 
number  is  multiplied  by  the  ratio  of  the  bandwidth  in  Hertz  to  the  load  shunt  resistance  in  ohms.  For 
illustration  we  choose  a  1  MHz  bandwidth  and  a  1  kilohm  resistor.  This  yields  1.66  x  10 17  A2  for  the 
thermal  noise  current  squared. 

A  typical  detector  dark  current  noise  of  1  nA  will  be  used  for  numerical  illustration.  The  dark  current 
noise  squared  is  3.2  x  10 19  times  the  dark  current  times  the  bandwidth.  Using  1  nA  for  the  vendor 
specified  dark  current  and  1  MHz  for  the  bandwidth  results  in  3.2  x  10*22  A2  for  the  dark  current  noise 
squared. 

The  shot  noise  current  squared  is  3.2  x  10 13  times  the  responsivity  times  the  signal  power  times  the 
electronic  bandwidth.  Our  assumed  responsivity  is  0.8  mA/mW  and  the  laser  output  power  is  8  mW. 
Again,  with  a  1  MHz  bandwidth,  the  shot  noise  current  squared  is  2  x  10  15  A2. 

It  is  typical  that  shot  noise  dominates  the  noise  for  the  detector  and  our  assumed  parameters  illustrate 
that  condition.  The  noise  current  squared  numbers  obtained  as  examples  are  added  to  get  the  total 
noise,  which  will  be  approximately  equal  to  just  the  shot  noise.  The  total  detector  noise  current  is  45 
nA.  Converted  to  equivalent  wavelength  by  using  5  picometer/milliampere,  the  detector  noise  is 
equivalent  to  2.2  xlO*4  picom  eters. 

Polarization  noise  results  from  the  source  varying  the  output  polarization.  The  reflections  (and  perhaps 
some  parameters)  inside  the  micro  resonator  depend  on  polarization,  so  the  variations  in  the 
polarizations  yield  different  losses  inside  the  resonator.  Consequently  the  power  measured  by  the 
photodiode  may  change  if  the  polarization  changes. 


Uncertainty  in  the  wavelength  separation  of  the  two  dips,  one  before  stimulus  and  the  other  after 
stimulus,  involves  sampling  and  other  previously  discussed  uncertainties.  There  are  at  least  two  effects 
that  interact  to  affect  the  separation  uncertainty.  The  laser  linewidth  is  a  fundamental  uncertainty  that 
limits  accuracy.  The  Agere  2300  laser  diode  previously  mentioned  has  a  linewidth  of  10  MHz.  This 
translates  to  a  wavelength  uncertainty  of  approximately  0.06  picometers  at  Aq  =  1.312  ^m  wavelength. 

In  effect,  this  is  the  width  of  the  tool  used  to  measure  the  resonant  dips,  so  its  width  is  a  limit  on 
defining  the  shape  of  a  resonance  dip.  The  Q  of  the  cavity  describes  the  width  of  the  resonance  dip.  See 
Appendix  B  for  the  resonant  linewidth  model.  In  terms  of  wavelength,  the  Q  is  equivalent  to  AA/Aq 
where  AA  is  the  full  width  at  half  maximum  and  A<>is  the  resonant  wavelength.  See  references  3 
through  5  for  more  detailed  discussions.  For  Q  =  106  the  resonant  linewidth  is  1.3  picometers.  That 
means  the  maximum  number  of  uncorrelated  samples  within  the  full  width  half  maximum  resonance 
width  is  1.3/0.06  or  approximately  22  samples.  However,  because  of  the  scanning  the  temporal 
sampling  rate  must  also  be  considered.  It  is  the  second  of  the  two  effects  under  discussion. 

The  temporal  sampling  rate  affects  a  number  of  performance  issues  in  addition  to  the  uncertainty.  Each 
ramp  can  be  thought  of  as  a  "sample"  unto  itself  since  we  assume  a  detection  of  a  resonance  shift 
occurs  only  once  during  a  given  ramp.  For  example,  suppose  one  wants  to  check  for  a  shift  400  times 
per  second.  That  means  a  scan  or  ramp  must  complete  within  2.5  milliseconds.  If  the  stimulus  is 
changing  with  time,  the  2.5  millisecond  separation  must  be  satisfactory  for  the  intended  application. 

The  next  issue  affecting  temporal  sampling  rates  is  the  range  of  wavelengths  to  be  search  for  the 
resonance  shift.  The  wavelength  range  can  be  guided  by  the  free  spectral  range  of  the  resonator.  For 
example,  consider  a  100  pm  diameter  sphere.  The  wavelength  that  resonates  in  the  sphere  must  have 
an  integral  number  of  wavelengths  around  the  circumference.  With  PMMA  index  of  refraction  is  1.492 
and  with  PDMS  the  index  is  1.4.  Near  1.312  pm  wavelength,  the  free  spectral  range  is  slightly  less  than  4 
nm  for  either  material  (  3.7  nm  for  PMMA  or  3.9  nm  for  PDMS.  Let  us  choose  the  laser  linewidth  of  0.06 
pm  to  be  our  sampling  distance.  To  span  4  nm,  we  need  67,000  samples.  With  400  scans  per  second, 
our  temporal  sampling  rate  must  be  approximately  27  M  samples  per  second. 

The  choice  of  the  laser  linewidth  as  the  sample  spacing  is  somewhat  arbitrary.  A  larger  sample  spacing 
provides  fewer  data  points  so  the  resonance  line  shape  is  not  as  well  resolved.  From  an  uncertainty 
viewpoint  the  laser  linewidth  is  already  an  uncertainty  whereas  the  sample  spacing  must  be  converted 
to  uncertainty  as  was  done  for  the  amplitude  digitization.  The  two  effects  are  independent  so  the  net 
uncertainty  is  the  square  root  of  the  sum  of  the  squared  individual  uncertainties.  If  we  choose  the 
sample  spacing  of  0.06  pm,  the  sampling  uncertainty  is  0.017  pm.  This  yields  a  net  uncertainty  from 
these  two  effects  of  0.062  pm.  The  laser  linewidth  dominates  the  sampling  uncertainty. 

5.6  Information  Rate 

The  microresonator  sensor  has  a  one  dimensional  signal  output  which  is  the  detector  output  as  a 
function  of  wavelength,  although  other  equivalent  variables  can  be  used.  As  shown  above  the  driving 
current  for  the  laser  diode  can  be  used  instead  of  wavelength  shift,  or  time  during  the  ramp  can  also  be 
used.  Consequently,  the  information  rate  is  modeled  by  the  following  equation. 
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H  is  the  information  rates  in  bits  per  measurement  (with  log  implying  base  2  logarithm);  S  is  the  signal 
power  spectral  density  (including  noise,  in  general)  and  N  is  the  noise  power  spectral  density.  £;  is  the 
frequency  so  that  B  is  the  bandwidth  (single  sided).  ^  and  B  have  appropriate  units,  e.g.  cycles  per  m 
with  our  choice  of  wavelength  as  the  independent  variable  for  the  signal. 

5.6.1  Information  Rate  for  Detection  of  Dip 

There  are  various  features  that  can  be  extracted  from  the  data.  One  of  these  is  simply  detecting  the 
dips  and  noting  the  time  separation  of  the  minima.  The  signal  for  this  feature  extraction  is  modeled  as 
follows.  The  basic  signal  prior  to  signal  processing  is  the  detector  output.  The  detector  output  (in  amps) 
is  linearly  related  to  the  optical  power  from  the  fiber.  The  laser  input  to  the  fiber  is  an  increasing  power 
over  time  resulting  from  the  current  ramp  driving  the  laser  diode.  As  the  current  changes  along  the 
ramp  the  output  wavelength  changes,  with  5  picometers  per  milliamp  being  our  example  value.  The 
power  output  dips  in  the  vicinity  of  a  resonance  of  the  micro  resonator.  The  width  and  shape  of  the  dip 
are  discussed  in  Appendix  B. 


The  first  step  in  processing  is  to  remove  the  unimportant  ramp  in  the  detector  output.  After  that,  the 
remaining  feature  is  the  sharp  dip  denoting  the  resonance.  There  can  be  more  than  one  resonance 
within  each  scan  of  wavelengths.  However  for  signal  spectrum  purposes,  we  assume  a  single  dip  before 
the  stimulus  is  applied  and  a  single  dip  afterthe  stimulus  is  applied.  We  assume  the  dips  are  identical 
except  for  their  locations  in  wavelength.  The  signal  power  is  then  the  square  of  the  detector  output  in 
amps  to  yield  a  power  signal  in  A2,  consistent  with  our  prior  noise  power  calculations. 


The  same  signal  to  noise  ratio  holds  if  the  dips  are  considered  pulses  with  peaks  instead  of  dips. 
Therefore,  we  use  the  following  as  our  signal. 
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MA  is  the  peak  of  the  current  signal  in  amps,  which  is  the  fractional  dip  times  the  responsivity  times  the 
optical  power.  The  constant  A  will  adjust  the  amplitude  to  the  proper  value  and  the  proper  units. 


We  need  the  transform  of  the  signal  squared.  Recall  that  the  transform  of  the  signal  magnitude  squared 
is  the  cross  correlation  of  the  transform  with  its  complex  conjugate.  The  transform  is  given  by  the 
following. 
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The  signal  power  spectrum  is  then  as  follows. 
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For  a  single  sided  power  spectrum  the  symmetry  of  the  above  power  spectrum  yields  the  following 
single  sided  expression. 


(5.23) 
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The  noise  power  spectrum  is  the  denominator  in  Eqn  (5.18).  Often  the  shot  noise  from  the  detection 
process  dominates  the  noise. 

5.6.2  Information  Rate  for  Detection  of  Dip:  A  Numerical  Example 

We  use  Eqn  (5.19)  with  some  assumptions  to  provide  a  numerical  example  of  the  information  rate  when 
detection  of  a  dip  is  the  goal.  Our  prior  example  values  are  used  except  that  we  increase  the  temporal 
bandwidth  consistent  with  a  27  million  samples  per  second  sampling  rate.  We  assume  the  detector  shot 
noise  is  the  noise  power  spectrum. 

The  signal  power  spectrum  is  given  in  Eqn  (5.23)  and  the  shot  noise  spectrum  has  that  same  spectral 
shape.  Thus  the  ratio  of  the  two  spectra  is  constant.  As  the  spectra  approach  zero,  other  noise  terms 
come  into  play.  We  choose  to  limit  the  spectrum  to  a  band  where  the  two  spectra  are  significant 
compared  to  zero.  For  integration  purposes  for  Eqn (5. 19)  we  are  integrating  a  constant  from  zero  out  to 
a  band  limit.  The  following  numerical  results  illustrate  the  process. 

The  peak  signal  current  arises  from  8  mW  of  optical  power,  converted  to  current  by  a  responsivity  of  0.8 
mA/mW,  and  a  fractional  peak  of  0.6,  or  3.84  milliamps.  This  yields  a  peak  squared  signal  current  of  4.1 
x  10  5  amps2.  The  resulting  shot  noise  current  squared  is  2(1.6  x  10'19)(3.84  x  10'3)  Af.  We  have  chosen 
the  electronic  bandwidth  to  be  50  MHz.  Note  that  the  electronic  bandwidth  is  not  the  bandwidth  for 
the  integration  in  Eqn  (5.19).  With  this  electronic  bandwidth  the  shot  noise  current  squared  is  equal  to 
6.14  x  10'14  amps2. 


In  Eqn  (5.19)  the  signal  spectrum  is  assumed  to  include  additive  noise.  The  integrand  is  then  log2(l+S/N) 
where  S  is  signal  only,  and  for  our  example  is  equal  to  29.3.  This  large  number  results  from  the  small 


noise  with  a  modest  optical  power,  in  essence,  the  information  rate  is  driven  by  the  number  of 
equivalent  noise  levels  found  in  the  signal,  hence  the  large  value. 

The  bandwidth  for  the  integral  is  the  final  issue  to  be  considered.  We  have  used  wavelength  as  the 
independent  variable  for  our  signal,  so  our  frequency  variable  ^  has  units  of  m*1  or  some  equivalent 
convenient  unit.  The  decaying  exponential  in  Eqn  (5.23)  will  be  considered  zero  when  the  following 
bandwidth  is  reached. 


3 

2nAA 


(5.24) 


AA  is  found  from  the  Q  of  the  resonator  (assume  Q=  106)  and  the  laser  wavelength  of  1.3  pm.  The  result 
is  A^  =  0.36  cycles  per  picometer. 


The  information  rate  is  then  10.7  bits  per  picometer  per  measurement. 

Intuitively  this  information  rate  suggests  one  can  resolve  roughly  1000  positions  within  a  picometer. 

This  is  true  if  one  considers  the  shot  noise  as  the  limiting  factor.  A  more  realistic  approach  is  to  use  the 
total  uncertainty  expressed  in  picometers.  As  an  example  the  uncertainty  caused  by  the  laser  linewidth 
is  a  larger  uncertainty  (~  0.06  pm)  than  the  noise  (-0.0012  pm,  a  factor  of  50  smaller).  With  the  laser 
linewidth  uncertainty  we  would  expect  to  resolve  approximately  16  spots  within  1  pm.  This  is 
approximately  4  bits  per  picometer  per  measurement.  This  estimate  does  not  account  for  the  different 
spectral  shape  of  the  laser  linewidth  uncertainty,  but  the  estimate  provides  further  intuitive  meaning  for 
what  the  information  rate  means  physically. 

5.6.3  Using  Cross  Correlation  to  Determine  Separation  of  Dips 

The  actual  signal  processing  consists  of  measuring  the  shift  in  the  resonant  frequency  rather  than  just 
detecting  the  presence  of  a  dip.  One  technique  for  determining  the  shift  is  to  locate  the  peak  value  in 
the  cross  correlation  of  the  before  stimulus  signal  and  the  after  stimulus  signal.  To  proceed  with  a 
numerical  example,  the  details  of  the  cross  correlation  of  two  Lorentzian  functions  is  presented  in 
Appendix  C.  The  before  stimulus  signal  and  the  after  stimulus  signal  are  assumed  to  be  the  same  except 
for  the  shift  in  the  resonant  wavelength.  While  the  actual  signal  is  a  dip,  we  first  focus  on  just  the  part 
that  varies  with  wavelengths.  Then  the  constants  will  be  considered.  From  the  appendices  the  two 
signals  are  given  by  the  following. 
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Their  cross  correlation  is  then  expressed  as  follows. 


M2A2x(U)’ _ 1_ _  (5  2 

4  [j-W-a)]!+(ai)j 

6  is  the  shift  or  lag  independent  variable  for  the  cross  correlation,  C.  AA  is  the  full  width  half  maximum 
of  the  lineshape.  Usually  this  is  estimated  from  the  Q  of  the  resonator  and  the  operating  wavelength 
given  the  following. 


M  is  the  fractional  depth  of  the  dip  that  is  normalized  to  be  between  0  and  1.  So  our  normalized  signals 
are  just  a  constant  -  g(A)  and  the  same  constant  -  gi(A).  The  cross  correlations  of  these  two  signals  then 
adds  the  square  of  the  constant  minus  the  mean  values  of  g  and  gi  to  C(6).  Under  our  assumptions  of 
identical  lineshapes ,  the  mean  values  of  g  and  g:  are  the  same,  and  are  equal  to  MAti(AA/2).  The 
constant  A  will  be  adjusted  to  make  the  peak  of  the  cross  correlation  equal  to  the  peak  signal  current 
squared. 

In  locating  the  dip  what  matters  is  the  change  in  the  signal  as  we  move  one  or  more  samples  away  from 
the  dip.  Therefore  we  get  that  information  by  working  from  Eqn  (5.26)  as  a  peaked  function.  The 
constant  additive  terms  discussed  above  do  not  affect  the  result.  If  our  change  in  signal  is  smaller  than 
the  noise,  the  noise  can  cause  an  uncertainty  in  location. 

The  sample  spacing  relative  to  the  width  AA  is  critical  in  determining  the  accuracy  of  peak  location. 

Using  our  example  numbers  the  linewidth  of  the  laser  limits  our  resolution  to  something  greater  than 
0.06  pm  whereas  the  FWHM  of  the  resonator  line  shape  function  is  approximately  1.3  pm.  With  an 
adequate  temporal  sample  rate  (27  M  samples  per  second)  the  lineshape  function  is  reasonably  well 
sampled.  There  will  be  approximately  22  samples  within  the  FWHM  portion  of  the  line  shape.  We  will 
treat  this  well  sampled  condition  and  not  consider  a  poorly  sampled  case. 

There  is  always  a  phasing  or  synchronization  issue  as  to  whether  the  peak  is  centered  within  a  sample. 
We  will  make  that  assumption  for  our  calculations.  Therefore,  the  sample  at  the  peak  itself  extends 
0.03  pm  to  either  side  of  the  peak.  The  adjacent  sample  is  centered  0.06  pm  from  the  peak  itself.  As 
one  moves  from  the  peak  sample,  the  adjacent  samples  change  by  factors  of  0.0021,  0.0083,  0.0185, 
0.0497,  etc.  That  means  that  the  noise  or  uncertainty  has  to  exceed  0.21  percent  of  the  peak  value  to 
cause  a  one  sample  uncertainty  in  peak  location.  Continuing  with  our  example  numbers  the  peak 
current  from  the  detector  is  8  mW  of  optical  power  times  a  responsivity  of  0.8  mA/mW  or  6.4  mA.  The 
resonance  will  dip  below  this  by  the  fraction  factor  M  for  which  we  have  used  0.6  as  an  example.  That 
means  at  the  depth  of  the  dip  the  current  is  0.4  times  6.4  or  2.56  mA.  The  uncertainty  must  cause  a  rise 
of  0.0021  time  2.56  mA  or  0.0269  pm  (using  5  pm/mA  for  wavelength  vs.  current  slope  for  laser). 


The  shot  noise  associated  with  a  signal  of  2.56  mA  is  4.1  x  10'14  A2.  This  is  equivalent  to  a  current  of  2.02 
x  10'A  mA,  which  is  0.001  pm.  The  shot  noise  is  approximately  4  %  of  the  signal  level  change  in  moving 
from  the  minimum  to  an  adjacent  sample. 

A  process  has  been  presented  and  illustrated  regarding  uncertainty  in  locating  a  dip  using  cross 
correlation.  As  expected  with  the  large  signal  level  and  the  very  small  shot  noise,  the  shot  noise  does 
not  impact  the  sample  selection  for  the  minimum.  The  shot  noise  is  only  about  4%  of  what  would  cause 
a  single  sample  uncertainty  in  the  minimum  location. 
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Equation  Section  lAppendixA  Numerical  Integration 

Eqn  (5.10)  can  be  expressed  in  different  algebraic  forms,  so  a  different  form  was  chosen.  Basically  it  is 
obtained  by  letting  cos0  be  x  and  replacing  cos2  80  by  (a2-a02)/a2.  The  end  result  is  the  following 
expression  for  the  Hn  coefficients. 


(A.1) 


Only  even  values  of  n  will  be  used,  so  one  can  use  2n  as  the  subscript  of  H  and  for  the  order  of  P,  the 
Legendre  polynomial.  In  that  case,  for  loops  one  uses  an  increment  that  advances  by  1.  Here,  we  use  an 
index  that  begins  with  0  and  increments  by  2.  Using  Matlab,  one  cannot  use  a  subscript  of  0  for  an 
array,  so  we  used  nn  =  n+1  where  n  advances  by  an  increment  of  2. 


The  convergence  of  the  integral  towards  zero  is  slow,  so  that  the  power  of  x  in  the  Legendre  polynomials 
gets  high.  This  can  create  numerical  noise  without  a  robust  algorithm  for  generating  the  Legendre 


polynomials.  Consequently,  the  built  in  function  legendre.m  was  used  and  was  successful,  legndre.m 
generates  the  associated  Legendre  polynomials  recognized  as  Pnm(x).  Pn(x)  is  the  m=0  associated 
Legendre  polynomial,  which  is  the  Matlab  subscript  1  array  from  the  m=0  to  m=n-l  associated  arrays 
generated.  The  code  snippet  below  illustrates  the  approach,  where  x  is  an  array  of  values  from  cos0o  to 
1.  Note  that  odd  orders  are  also  generated  even  though  they  are  not  used.  The  1001  vectors  contain 
the  Pn(x)  values  needed  for  the  integration. 

%%%%%%%%%%%%%%%%%%%%%%%%  Legendre  Polynomials 
^Generate  the  Legendre  Polynomials.  Use  legendre  which  generates  the 
%associated  Legendre  Polynomials,  Psub  n  superscript  m  (x).  The  m=0 
%(Matlab  subscript  of  l)array  is  the  ordinary  Legendre  Polynomials 
for  n=0:l:1000  ^Generate  more  than  twice  the  n  loop  limit  below  since 
%this  function  uses  a  backward  recursion  algorithm.  That  is,  both 
%even  and  odd  orders  are  generated.  The  code  below  outside  this  loop 
^selects  the  even  orders.  Start  with  n=0  to  get  P  sub  0  array.  Note  that 
%P  starts  with  an  index  of  1,  which  is  P  sub  0  array. 

Ptemp=legendre(n,x);  %Generates  the  associated  Leg  Poly's 
P(n+l,:)=Ptemp(l,:);  ^selects  the  m  =  0  array  to  keep 
clear  Ptemp 
end  %end  n  loop 

The  integral  from  Eqn  (A.l)  was  accomplished  using  a  Simpson's  rule  approach,  modified  slightly  to 
eliminate  a  trivial  imaginary  part  to  the  first  value  in  the  integration.  The  code  below  creates  401  even 
order  values  for  the  integral.  This  will  subsequently  be  multiplied  by  the  term  outside  the  integral  in  Eqn 
(A.l).  This  code  uses  an  x  array  described  earlier,  the  sphere  radius  a  and  a  calculated  radius  of  the  rigid 
plate  contact  area. 

%%%%%%%%%%%%  Integral  Calculation  %%%%%%%%%%%%%%%% 

for  nn=l:401 

intgrnd(nn,:)=sqrt((x(l,:).A2)*aA2-(aA2-aOA2)).*P(2*nn,:); 

%The  true  Simpson's  rule  is  commented  out  because  the  term 
%int(nn,l)  causes  a  complex  result.  Removing  it  doesn't  change  the 
%values  and  the  imaginary  part  was  very  small.  I  chose  to  use  the 
%magnitude  or  abs(int(nn,l),  but  it  doesn't  really  matter. 
integral(nn)=  dx/3*(abs(intgrnd(nn,l))+  2*sum(intgrnd(nn,3:2:end-2))... 

+  4  *sum(intgrnd(nn,2:2:end))+  intgrnd(nn,end)); 

%integral(nn)=  dx/3*(intgrnd(nn,l)+2*sum(intgrnd(nn,3:2:end-2))... 
%+4*sum(intgrnd(nn,2:2:end))+intgrnd(nn,end)); 
end  %end  nn  loop 


Equation  Section  lAppendix  B  Model  for  Resonance  Dip 


A  number  of  authors  state  that  the  resonance  dip  is  well  approximated  by  a  Lorentzian  lineshape.  There 
are  a  variety  of  different  expressions  used  for  this  line  shape  that  depend  on  normalization  mostly.  We 
choose  to  work  in  wavelengths  (rather  than  frequency),  so  our  expression  follows  in  Eqn  (B.l). 
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(B.l) 


A  is  a  constant  that  will  be  adjusted  to  make  the  peak  power  correct  and  to  have  correct  units.  In  this 
form  M  represents  the  fractional  peak  value  or  the  depth  of  the  dip  if  we  subtract  g(A)  from  1  or  the 
constant  signal.  That  is,  evaluating  at  Aq  yields  M  times  A.  M  has  a  value  between  0  and  1.  AA  is  the  full 
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width  at  half  maximum  as  can  be  seen  by  evaluating  g(A)  at  Z  =  /L  ± - .  The  total  area  under  the 

*  2 


function  in  this  form  is  7t  times  AA/2  times  M  times  A. 


The  parameters  are  measured  from  data  as  follows.  The  resonant  wavelength  Ao  is  found  by  locating  the 
minimum  value  of  the  transmitted  power.  The  ramp  resulting  from  the  current  ramp  can  be  removed 
from  the  signal  priorto  processing.  AA  is  measured  as  the  full  width  at  half  max  on  the  dip.  M  is  the 
depth  of  the  dip. 


There  are  other  quantities  that  relate  to  these  parameters.  For  example,  the  Q  of  the  system  is  equal  to 
AA/Ao.  Naweed  et  al.  [5]  give  the  following  expression  for  M. 
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T  represents  the  coupling  transmission  from  the  fiber  to  the  sphere  and  a 


is  the  loss  in  the  sphere. 


Unfortunately,  it  is  not  easy  to  separate  these  two  loss  quantities  from  usual  data.  Consequently  Eqn 
(B.2)  is  offered  for  theoretical  interest.  Notice  however  that  the  depth  increases  (1  -  M  gets  smaller) 

with  increasing  T and  decreases  with  increases  in  at 


Equation  Chapter  3  Section  lAppendix  C  Crosscorrelation  of  Lorentzian 
Lineshape  Functions 

The  Fourier  transform  of  e~^  is  given  by  the  following  expression. 


F[e'2^]  =  J“  e"J*H e~j2x(xdx  =f  e^e'^dx  +  J  V^V'2*^ 


(C.l) 


By  expanding  the  exponential  with  an  imaginary  argument  into  cosines  and  sines,  then  changing  the 
dummy  variable  of  integration  on  the  first  integral  to  a  =  -  x,  one  obtains  the  following,  after  consulting 
a  table  of  integrals. 


F{e*™}  =  cos{2K^a)exp(-2Kj3a)da 
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P  is  a  constant  and  the  independent  variable  is  ^  This  Lorentzian  function  and  its  transform  can  be  used 
to  obtain  the  cross  correlation  of  the  lineshape  functions  of  interest. 


We  use  the  same  functional  shape  before  stimulus  and  after  stimulus  except  for  a  shift  of  the 
resonance.  Eqn  (B.l)  describes  the  resonance  lineshapes  that  we  will  use.  Without  loss  of  generality  we 
set  X0  to  0  as  the  before  stimulus  response.  The  after  stimulus  response  will  shift  the  resonance  to  X1  so 
our  second  lineshape  is  also  obtained  from  Eqn  (B.l)  with  Xq  replaced  by  The  term  inside  the 
brackets  below  matches  the  form  of  Eqn  (C.2). 


g]  {X)=MAk^y 
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In  the  transform  domain,  the  product  of  the  two  functions  is  the  following. 
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The  bracketed  terms  show  the  individual  functions  in  the  transform  domain.  Combining  the  two 
identical  decaying  exponential  terms  and  bringing  the  result  out  of  the  transform  domain  yields  the 
cross  correlation,  C(6).. 
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Generalizing  to  a  Ao  not  zero,  replace  Xx  by  Ai  -  A 0.  Some  liberty  was  taken  with  signs  since  the  cross 
correlation  of  these  even  functions  is  an  even  function.  Somewhat  as  expected  the  cross  correlation  is  a 
Lorentzian  function  with  twice  the  FWHM  value  of  the  individual  lineshapes.  We  finally  write  the 
following  for  our  basic  signal  arising  from  the  cross  correlation.  While  the  signals  actually  dip,  the  noise 
and  uncertainty  and  SNR  will  be  the  same  if  we  use  these  peaked  functions. 
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The  constant  A  will  be  adjusted  so  that  the  peak  value  is  the  signal  current  squared  in  amps  squared. 

For  example,  with  8  mW  of  optical  power,  detector  responsivity  of  0.8  mA/mW,  A  will  be  6.4  mA  times  2 
divided  by  the  square  root  of  pi  times  AA  with  units  of  mA  per  m  to  the  one  half  power. 
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Profile  Sensing  Using 

Eddie  L.  Jacobs 


Sensor  Atoms 


Abstract — The  concept  of  a  sensor  atom  as  a  fundamen¬ 
tal  unit  of  sensing  Is  Introduced.  The  properties  of  one 
particular  type  of  sensor  atom,  the  Bernoulli  sensor  atom, 
are  derived. 

I.  Introduction 

Profiling  sensors  have  been  demonstrated  to  provide 
sufficient  information  about  a  moving  object  to  perform 
many  useful  classifications  such  as  whether  or  not  the 
object  is  a  human,  an  animal,  or  a  vehicle.  A  profiling 
sensor  extracts  basic  shape  information  about  a  moving 
object  and  classifies  it  into  two  or  more  categories. 
These  types  of  sensors  could  be  a  low  cost  alternative 
to  imaging  sensors  for  applications  involving  border  and 
perimeter  security.  To  date,  they  sensors  have  been  built 
using  linear  arrays  of  sensors.  The  linear  arrays  have 
either  been  a  single  integrated  component  or  collections 
of  several  discrete  components. 

Atoms  are  the  fundamental  unit  of  matter.  Atoms  in 
combination  form  all  the  stuff  we  see.  A  large  part 
of  modem  physics  has  been  devoted  to  studying  the 
properties  of  atoms  and  how  they  interact  to  form 
molecules.  The  concept  of  a  sensor  atom  presented  in 
this  research  is  an  attempt  to  leverage  some  of  the  prin¬ 
ciples  of  modem  atomic  physics  and  apply  them  to  the 
problem  of  sensor  design  with  particular  application  to 
the  profiling  sensor.  The  main  result  of  this  research  will 
be  a  software/hardware  infrastructure  that  facilitates  the 
development  of  complex  sensing  systems  from  simple 
elements. 


Sensor  atoms  are  sensors.  Hence,  the  properties  of 
sensors  are  the  properties  of  sensor  atoms.  In  general 
sensors  sense  some  sensible  quantity  and  output  some 
other  sensible  quantity.  For  instance,  a  typical  electro¬ 
optic  detector  senses  light  and  outputs  a  current  or 
voltage.  Sensible  quantities  can  be  described  in  terms 
of  domain  modes.  Domain  modes  are  functions  that 
describe  the  behavior  of  the  sensor  with  respect  to  some 
domain.  As  an  example,  an  electo-optical  sensor  may 
only  sense  energy  from  a  certain  angular  region  of  space 
and  over  a  limited  spectral  bandwidth.  As  a  result,  an 
electro-optical  sensor  will  have  a  spatial  domain  mode 
(or  simply  spatial  mode)  and  a  spectral  mode.  The  sensor 
may  be  more  sensitive  to  one  polarization  of  light  than 
another  and  will  have  a  polarization  mode.  It  will  also 
be  limited  in  how  fast  it  can  respond  to  changes  in  light 
energy1  and  will  have  a  temporal  mode.  These  modes 
may  be  independent  or  may  be  coupled.  For  instance, 
if  a  sensor  has  no  sensitivity  to  polarization,  the  spatial 
and  polarization  modes  will  be  independent  On  the  other 
hand,  as  is  often  the  case,  polarization  sensitivity  may 
vary  spatially  and  as  a  result  the  polarization  and  spatial 
modes  will  be  coupled. 

Within  an  element  atoms  are  described  fairly  simply 
and  all  are  the  same.  However,  the  properties  of  a 
material  are  more  strongly  determined  by  the  configu¬ 
ration  of  the  atoms  rather  than  the  atoms  themselves. 

]The  temporal  mode  is  restricted  to  time  variations  that  are  slow 
with  respect  to  the  mean  frequency  of  the  light. 
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Carbon  atoms  arranged  in  one  way  form  the  lead  in  a 
pencil.  Arranged  another  way  they  form  a  diamond.  Sim¬ 
ilarly,  sensor  atoms  when  combined  together  will  have 
properties  that  are  not  only  governed  by  the  individual 
properties  of  the  atoms  but  also  by  the  rules  by  which 
they  interact  Rules  established  for  the  interaction  of 
sensor  atoms  presume  the  existence  of  some  mechanism 
for  facilitating  the  interaction  such  as  a  communication 
channel  and  protocol.  This  mechanism  is  also  properly 
a  property  of  the  sensor  atom. 

As  a  simple  example  of  the  interaction  of  sensor 
atoms,  suppose  two  simple  break-beam  sensors  (atoms) 
are  positioned  horizontally  some  distance  apart.  These 
sensors  send  out  an  optical  beam  that  strikes  a  reflecting 
surface  and  returns  to  the  receiver  which  is  co-located 
with  the  beam  transmitter.  The  output  of  each  sensor  is 
one  of  two  states;  “OPEN”  if  the  beam  is  not  broken  and 
“CLOSED”  if  it  is.  As  an  object  passes  by  it  will  break 
one  beam  and  then  the  other.  If  the  distance  between 
the  sensors  is  sufficiently  short,  objects  of  interest  will 
at  some  time  be  breaking  both  beams  during  transit. 
Assume  each  sensor  atom  can  report  its  own  state  (open 
or  closed)  and  has  knowledge  of  the  state  of  other  sensor 
at  all  times.  Each  atom  now  has  enough  information  to 
report  something  more  than  just  whether  it  is  open  or 
closed.  A  proper  choice  of  interconnection  rules  of  the 
atoms  will  allow  each  atom  to  report  the  direction  of 
travel  of  the  object.  An  example  of  an  interconnection 
rule  would  be  that  if  a  sensor  (say  the  left  one)  changes 
state  from  open  to  closed  and  knows  that  the  right  sensor 
is  in  a  closed  state  then  the  object  is  traveling  from  right 
to  left  and  the  atom  will  report  that  the  direction  of  travel 
is  "LEFT".  This  is  something  more  than  either  sensor 
can  discern  alone  and  is  possible  due  to  the  interaction 
of  both  sensors. 

This  example,  while  trivial,  provides  some  insight 


into  the  power  of  interconnected  sensor  atoms.  Sensors 
of  minimal  complexity  joined  together  by  appropriate 
interconnection  rules  allow  sensors  to  be  developed  that 
are  robust,  optimal  for  their  intended  tasks,  and  make 
efficient  use  of  power  and  bandwidth  resources.  This 
research  explores  some  aspects  of  this  capability  us¬ 
ing  theoretically  simple  and  physically  realizable  sensor 
atoms.  The  remainder  of  the  paper  is  organized  as 
follows.  The  next  section  describes  a  particular  type  of 
sensor  atom  termed  the  Bernoulli  sensor  atom. 


II.  A  Bernoulli  Sensor  Atom 


A .  General  Theory 


Consider  a  simple  sensor  whose  functional  block 
diagram  is  as  shown  in  Figure  .  Input  to  the  sensor 
produces  a  sensed  signal  S  (t)  which  is  corrupted  by 
additive  noise  N  (t)  to  produce  the  signal  V  ( t )  which 
is  sampled  every  T  seconds  to  produce  the  random 
sequence  Vn  where  n  is  the  index  of  the  nth  sample. 
Each  value  of  Vn  is  compared  with  a  threshold  value  7 
to  produce  the  following  binary  sequence. 


B 


n  — 


vn  >7 

Vn  <7 


(1) 


The  sequence  Bn  is  a  Bernoulli  sequence  character¬ 
ized  to  first  order  by  the  probability  p  where 


P  —  P  [-Bn  =  l]  =  P  [Kj  >  7]  =■  P\V  (nT)  >  7]  (2) 
If  V  (: t )  is  a  stationary  random  signal,  then 

r+00 

P=  fv  (v)  dv  (3) 

Jy 

where  fv  ( v )  is  the  first  order  probability  density 
function  (PDF)  of  the  random  variable  V  (f).  Note  that 
because  of  its  binary  nature,  E  [£n]  =  p. 
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Tbc  second  order  statistics  of  Bn  are  described  by 
Rb  (*)  »  E[BnBn+k]  =  P{Bn  =  1,  Bn+k  =  1],  If 
Vi  =  V  ( nT )  and  V2  =  V  (nT  +  JfeT)  then 

/  +  oo  /**f  oo 

J  fvlvi(viiv2)dvidv2  (4) 

where  (^i,  ^2)  represents  the  joint  PDF  of  the  sig¬ 
nal  V  (t)  at  times  nT  and  nT+kT.  Further  development 
requires  knowledge  of  the  fir  st  and  second  order  densities 
of  V  (t). 


If  pv  =  0  and  7  =  0  then  Zn  is  the  output  of  a  classic 
hard  limiter  and  Rz  ( k )  =  ^  arcsin  yeilding 

Rb  ^  =  <tr  arCSin  Pk  +  \ 

.  For  more  general  conditions  equation  8  must  be  used. 


B.  The  Gaussian  Assumption 

If  V  ( t )  is  assumed  to  be  a  non-periodic  Gaussian 
process  with  autocorrelation  Rv  (r)  then 

fv(v)  =  G( v.  fiV,c TV)  =  [2iral]->  exp 
and 


fv: \Vi  (^1 ,  ^2)  = 


-1  -1 


27rc7v\/l  —  ~p\  exp  {—  [2cr^,  (l  —  yo^)]  1  [(vi  -  Rvf  -  2pk  (vi  -  rv)  (v2  -  Rv)  +  (t>2  -  AH')2] } 

(6) 


where  py  =  lim  Ry  (r),  <Jy+py  =  Ry  (0),  and 

T  — ►  OO 

pk  =  It  is  convenient  to  re-wnte  equation 

6  using  the  definition  of  conditional  probability  as 


Mv,  (^1,^2)  =  fv j  (^2)  fVi/Vi  (t>i)  =  fv  MfVi/V,  (vi) 

(7) 

where  fVl/v,(v l)  =  G(vi  '■  Rc(fl)i^c)  with 
Rc  (vi)  -  Rv  +  Pk  fai  “  Rv)  and  ac  =  cv^/l  -  p\. 
Using  this  definition  then  equations  3  and  4  can  be 
written  in  terms  of  the  cumulative  normal  function  or 
P=  !-*(“)  and 


Rb  ( k )  =P-  J*°°  fv  (v)  $  (7  £(v) 

with  $  (x)  =  A  useful  particular 

result  can  be  obtained  by  defining  the  auxiliary  random 
variable  Zn  =  2Bn  —  1.  Then  Rb  (k)  =  \Rz  (k)+p— 


j  dv  (8) 


